Jul 28 01:25:52.887792 ip-10-0-133-123 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 01:25:53.243841 ip-10-0-133-123 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:53.243841 ip-10-0-133-123 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 01:25:53.243841 ip-10-0-133-123 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:53.243841 ip-10-0-133-123 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 01:25:53.243841 ip-10-0-133-123 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:53.245055 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.244980    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 01:25:53.251146 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251131    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:53.251146 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251146    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251151    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251154    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251158    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251161    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251163    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251166    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251169    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251172    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251174    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251177    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251179    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251182    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251185    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251187    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251190    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251193    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251195    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251198    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251200    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:53.251210 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251203    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251206    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251208    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251211    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251213    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251216    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251219    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251222    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251224    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251227    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251229    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251231    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251234    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251237    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251239    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251241    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251244    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251246    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251249    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251251    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:53.251686 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251261    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251264    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251266    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251269    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251271    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251274    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251278    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251281    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251284    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251287    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251289    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251292    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251294    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251297    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251300    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251303    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251305    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251309    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251311    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:53.252270 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251314    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251316    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251318    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251321    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251325    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251329    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251331    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251334    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251337    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251340    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251344    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251346    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251349    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251351    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251354    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251358    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251361    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251364    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251366    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251369    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:53.252746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251372    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251375    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251377    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251379    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251382    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251384    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251387    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251389    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251392    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251394    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251396    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251398    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251401    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251403    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251405    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251408    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251411    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251414    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.251416    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252091    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:53.253206 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252097    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252100    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252103    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252106    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252108    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252111    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252113    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252116    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252119    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252122    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252125    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252128    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252130    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252133    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252135    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252138    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252140    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252143    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252147    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:53.253702 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252150    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252153    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252156    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252158    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252161    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252163    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252166    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252168    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252170    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252173    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252175    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252178    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252181    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252183    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252186    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252188    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252190    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252193    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252195    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252197    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:53.254158 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252200    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252203    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252209    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252211    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252214    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252216    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252219    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252221    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252224    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252227    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252231    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252234    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252236    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252240    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252242    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252245    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252247    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252250    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252252    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252255    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:53.254650 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252257    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252260    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252262    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252265    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252267    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252270    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252272    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252275    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252277    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252280    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252282    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252285    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252288    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252291    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252294    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252297    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252299    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252302    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252304    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252306    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:53.255138 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252310    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252313    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252316    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252319    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252321    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252324    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252326    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252328    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252331    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252333    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252335    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252338    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252340    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252343    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252345    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252348    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252350    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252353    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252355    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:53.255619 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252708    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252712    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252715    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252717    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252720    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252722    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252728    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252731    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252734    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252737    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252739    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252742    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252744    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252747    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252750    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252752    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252755    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252757    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252760    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252762    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:53.256095 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252765    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252767    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252769    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252772    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252774    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252777    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252780    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252782    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252785    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252787    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252790    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252792    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252794    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252797    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252799    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252801    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252804    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252807    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252811    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252813    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:53.256573 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252815    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252818    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252821    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252823    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252826    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252828    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252831    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252833    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252835    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252838    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252840    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252843    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252845    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252848    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252850    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252853    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252855    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252857    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252860    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252862    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:53.257027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252865    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252868    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252870    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252873    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252875    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252878    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252880    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252884    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252887    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252889    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252893    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252896    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252898    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252900    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252903    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252905    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252908    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252911    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252914    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:53.257578 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252916    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252918    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252921    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252923    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252926    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252928    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252930    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252933    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252935    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252937    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252940    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252942    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252945    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252949    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252952    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252956    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252959    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252961    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252964    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:53.258016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.252967    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253033    2555 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253039    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253045    2555 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253051    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253055    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253058    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253062    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253066    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253070    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253073    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253076    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253079    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253082    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253085    2555 flags.go:64] FLAG: --cgroup-root=""
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253087    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253090    2555 flags.go:64] FLAG: --client-ca-file=""
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253093    2555 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253096    2555 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253100    2555 flags.go:64] FLAG: --cluster-domain=""
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253103    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253106    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253108    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253112    2555 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 01:25:53.258459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253116    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253119    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253122    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253125    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253128    2555 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253131    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253133    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253136    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253139    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253143    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253146    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253149    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253153    2555 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253156    2555 flags.go:64] FLAG: --enable-server="true"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253159    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253163    2555 flags.go:64] FLAG: --event-burst="100"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253166    2555 flags.go:64] FLAG: --event-qps="50"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253169    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253172    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253176    2555 flags.go:64] FLAG: --eviction-hard=""
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253179    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253182    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253184    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253187    2555 flags.go:64] FLAG: --eviction-soft=""
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253190    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 01:25:53.259064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253192    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253195    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253198    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253201    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253203    2555 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253207    2555 flags.go:64] FLAG: --feature-gates=""
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253211    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253214    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253217    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253220    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253223    2555 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253226    2555 flags.go:64] FLAG: --help="false"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253229    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253232    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253235    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253237    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253241    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253244    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253246    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253250    2555 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253253    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253255    2555 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253258    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253261    2555 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 01:25:53.259692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253264    2555 flags.go:64] FLAG: --kube-reserved=""
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253267    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253270    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253273    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253276    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253278    2555 flags.go:64] FLAG: --lock-file=""
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253281    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253284    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253286    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253291    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253294    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253296    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253299    2555 flags.go:64] FLAG: --logging-format="text"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253303    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253306    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253308    2555 flags.go:64] FLAG: --manifest-url=""
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253311    2555 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253315    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253317    2555 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253321    2555 flags.go:64] FLAG: --max-pods="110"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253324    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253327    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253330    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253333    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253336    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 01:25:53.260269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253338    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253341    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253348    2555 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253351    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253354    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253357    2555 flags.go:64] FLAG: --pod-cidr=""
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253360    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253365    2555 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253367    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253370    2555 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253373    2555 flags.go:64] FLAG: --port="10250"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253376    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253379    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02e2e70af3e2a34e7"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253382    2555 flags.go:64] FLAG: --qos-reserved=""
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253385    2555 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253388    2555 flags.go:64] FLAG: --register-node="true"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253390    2555 flags.go:64] FLAG: --register-with-taints=""
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253394    2555 flags.go:64] FLAG: --registry-burst="10"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253396    2555 flags.go:64] FLAG: --registry-qps="5"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253399    2555 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253405    2555 flags.go:64] FLAG: --reserved-memory=""
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253409    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253412    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253414    2555 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253417    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 01:25:53.260860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253420    2555 flags.go:64] FLAG: --runonce="false"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253422    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253425    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253428    2555 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253431    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253434    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253437    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253440    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253443    2555 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253445    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253449    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253452    2555 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253454    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253457    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253460    2555 flags.go:64] FLAG: --system-cgroups=""
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253462    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253467    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253470    2555 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253473    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253477    2555 flags.go:64] FLAG: --tls-min-version=""
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253480    2555 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253482    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253485    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253488    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253491    2555 flags.go:64] FLAG: --v="2"
Jul 28 01:25:53.261447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253494    2555 flags.go:64] FLAG: --version="false"
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253498    2555 flags.go:64] FLAG: --vmodule=""
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253502    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.253505    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253636    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253639    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253642    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253645    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253647    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253650    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253653    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253656    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253659    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253663    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253666    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253669    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253672    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253676    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253679    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:53.262044 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253681    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253684    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253687    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253689    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253691    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253694    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253697    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253700    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253702    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253705    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253707    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253710    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253713    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253716    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253718    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253722    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253724    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253726    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253729    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253731    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:53.262541 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253734    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253736    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253738    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253741    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253743    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253746    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253748    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253751    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253753    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253756    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253759    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253761    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253763    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253766    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253768    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253771    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253773    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253776    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253778    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253780    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:53.263016 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253783    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253786    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253788    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253790    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253793    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253795    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253797    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253801    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253803    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253805    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253807    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253810    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253813    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253816    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253819    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253822    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253824    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253827    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253830    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:53.263477 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253833    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253835    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253838    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253840    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253844    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253846    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253849    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253851    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253854    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253856    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253858    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253860    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253863    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253865    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253868    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253871    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253873    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253875    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253877    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253880    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:53.263971 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253884    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:53.264454 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253886    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:53.264454 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253889    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:53.264454 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253891    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:53.264454 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.253893    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:53.264454 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.254455    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:53.264454 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.260971    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 01:25:53.264454 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.260985    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 01:25:53.264454 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.261004    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 01:25:53.264454 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.261009    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 01:25:53.264454 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261062    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:53.264454 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261067    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:53.264454 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261070    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:53.264454 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261073    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:53.264454 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261076    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:53.264454 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261078    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261081    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261084    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261086    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261090    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261094    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261097    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261099    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261102    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261105    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261107    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261111    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261114    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261117    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261120    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261123    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261125    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261129    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261131    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:53.264837 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261134    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261136    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261139    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261141    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261144    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261147    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261149    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261152    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261154    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261156    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261159    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261162    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261164    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261166    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261169    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261171    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261174    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261176    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261179    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:53.265277 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261181    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261183    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261186    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261188    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261191    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261193    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261195    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261199    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261201    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261203    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261206    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261208    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261211    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261214    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261216    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261219    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261221    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261224    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261226    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261229    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:53.265763 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261232    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261234    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261237    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261239    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261242    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261244    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261246    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261249    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261252    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261254    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261256    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261258    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261261    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261263    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261266    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261268    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261271    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261273    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261275    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261278    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:53.266243 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261280    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:53.266727 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261283    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:53.266727 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261285    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:53.266727 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261287    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:53.266727 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261289    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:53.266727 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261292    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:53.266727 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261295    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:53.266727 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261297    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:53.266727 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261299    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:53.266727 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261302    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:53.266727 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261304    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:53.266727 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261306    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:53.266727 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261309    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:53.266727 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261313    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:53.266727 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261316    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:53.266727 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261319    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:53.267079 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.261324    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:53.267079 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261431    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:53.267079 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261435    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:53.267079 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261439    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:53.267079 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261443    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:53.267079 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261446    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:53.267079 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261449    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:53.267079 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261452    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:53.267079 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261454    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:53.267079 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261457    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:53.267079 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261459    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:53.267079 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261462    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:53.267079 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261464    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:53.267079 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261467    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261469    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261472    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261474    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261476    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261479    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261481    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261483    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261486    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261488    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261490    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261493    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261495    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261498    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261500    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261503    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261506    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261508    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261511    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261513    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:53.267412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261515    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261531    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261534    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261537    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261539    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261542    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261544    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261546    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261549    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261551    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261554    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261556    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261558    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261562    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261565    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261568    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261571    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261573    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261576    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261578    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:53.267901 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261581    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261583    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261586    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261588    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261592    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261595    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261598    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261601    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261604    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261607    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261610    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261613    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261615    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261617    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261620    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261622    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261624    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261627    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261629    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261632    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:53.268412 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261634    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261636    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261639    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261642    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261644    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261646    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261649    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261651    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261654    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261656    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261659    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261661    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261664    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261666    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261669    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261671    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261673    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261676    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261678    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261681    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:53.268894 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261683    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:53.269365 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261685    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:53.269365 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261688    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:53.269365 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261691    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:53.269365 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261693    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:53.269365 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261696    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:53.269365 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:53.261698    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:53.269365 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.261702    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:53.269365 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.261794    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 01:25:53.269365 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.263607    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 01:25:53.269365 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.264814    2555 server.go:1019] "Starting client certificate rotation"
Jul 28 01:25:53.269365 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.264913    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 01:25:53.269365 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.264948    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 01:25:53.269365 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.267027    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 01:25:53.270466 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.270453    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 01:25:53.281308 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.281288    2555 log.go:25] "Validated CRI v1 runtime API"
Jul 28 01:25:53.283516 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.283497    2555 log.go:25] "Validated CRI v1 image API"
Jul 28 01:25:53.284905 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.284886    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 01:25:53.287256 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.287238    2555 fs.go:135] Filesystem UUIDs: map[132946e9-2d2a-4221-8f0c-0ffc0ede67c4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f54dbc47-9aeb-4822-b45f-040ea63f6430:/dev/nvme0n1p4]
Jul 28 01:25:53.287336 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.287254    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 01:25:53.291263 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.291161    2555 manager.go:216] Machine: {Timestamp:2026-07-28 01:25:53.290706815 +0000 UTC m=+0.300225240 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3202376 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec216d051940a59f4e7d85c2a4a854f9 SystemUUID:ec216d05-1940-a59f-4e7d-85c2a4a854f9 BootID:fc786dd2-9e97-40a2-9355-9e5102c7af2e Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:83:31:ee:cd:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:83:31:ee:cd:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:b7:1b:07:e9:bb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 01:25:53.291263 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.291256    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 01:25:53.291364 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.291322    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 01:25:53.292915 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.292900    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 01:25:53.293983 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.293955    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 01:25:53.294134 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.293987    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-123.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 01:25:53.294691 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.294664    2555 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 01:25:53.294721 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.294694    2555 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 01:25:53.294721 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.294702    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 01:25:53.295255 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.295244    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 01:25:53.296044 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.296034    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 01:25:53.298000 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.297989    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 01:25:53.298115 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.298105    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 01:25:53.298150 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.298143    2555 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 01:25:53.298184 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.298158    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 01:25:53.298184 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.298169    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 01:25:53.298184 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.298177    2555 kubelet.go:398] "Adding apiserver pod source"
Jul 28 01:25:53.298287 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.298186    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 01:25:53.298507 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.298493    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 01:25:53.298583 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.298515    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 01:25:53.300253 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.300224    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 01:25:53.301982 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.301962    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 01:25:53.302034 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.302010    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 01:25:53.303558 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.303544    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 01:25:53.303592 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.303566    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 01:25:53.303592 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.303574    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 01:25:53.303592 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.303579    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 01:25:53.303592 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.303588    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 01:25:53.303592 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.303593    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 01:25:53.303733 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.303601    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 01:25:53.303733 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.303606    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 01:25:53.303733 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.303612    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 01:25:53.303733 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.303626    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 01:25:53.303733 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.303637    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 01:25:53.303733 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.303642    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 01:25:53.303733 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.303666    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 01:25:53.303733 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.303671    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 01:25:53.307100 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.307079    2555 server.go:1268] "Started kubelet"
Jul 28 01:25:53.307280 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.307183    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 01:25:53.307280 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.307222    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 01:25:53.307388 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.307310    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 01:25:53.307852 ip-10-0-133-123 systemd[1]: Started Kubernetes Kubelet.
Jul 28 01:25:53.310542 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.310500    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 01:25:53.310630 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.310587    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 01:25:53.310702 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.310686    2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-123.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 01:25:53.311314 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.311291    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 01:25:53.311892 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.311872    2555 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 01:25:53.313358 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.313325    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 01:25:53.316592 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.315723    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-123.ec2.internal.18c64fb7d73347af  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-123.ec2.internal,UID:ip-10-0-133-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-123.ec2.internal,},FirstTimestamp:2026-07-28 01:25:53.307051951 +0000 UTC m=+0.316570367,LastTimestamp:2026-07-28 01:25:53.307051951 +0000 UTC m=+0.316570367,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-123.ec2.internal,}"
Jul 28 01:25:53.317412 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.317395    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 01:25:53.317412 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.317407    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 01:25:53.317556 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.317467    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-123.ec2.internal\" not found"
Jul 28 01:25:53.318081 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.318065    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 01:25:53.318160 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.318066    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 01:25:53.318160 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.318101    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 01:25:53.318239 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.318184    2555 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 01:25:53.318239 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.318195    2555 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 01:25:53.319165 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.319149    2555 factory.go:55] Registering systemd factory
Jul 28 01:25:53.319165 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.319165    2555 factory.go:223] Registration of the systemd container factory successfully
Jul 28 01:25:53.319383 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.319371    2555 factory.go:153] Registering CRI-O factory
Jul 28 01:25:53.319454 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.319386    2555 factory.go:223] Registration of the crio container factory successfully
Jul 28 01:25:53.319454 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.319433    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 01:25:53.319593 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.319457    2555 factory.go:103] Registering Raw factory
Jul 28 01:25:53.319593 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.319486    2555 manager.go:1196] Started watching for new ooms in manager
Jul 28 01:25:53.319935 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.319857    2555 manager.go:349] Starting recovery of all containers
Jul 28 01:25:53.322438 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.322284    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 01:25:53.322438 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.322342    2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 01:25:53.329874 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.329858    2555 manager.go:354] Recovery completed
Jul 28 01:25:53.331634 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.331616    2555 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 28 01:25:53.335205 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.335194    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:53.337353 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.337323    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:53.337439 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.337364    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:53.337439 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.337374    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:53.337863 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.337846    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 01:25:53.337863 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.337862    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 01:25:53.337986 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.337872    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:53.337986 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.337880    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 01:25:53.338607 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.338594    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:53.338685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.338622    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:53.338685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.338635    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:53.339365 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.339303    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-123.ec2.internal.18c64fb7d9019f8e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-123.ec2.internal,UID:ip-10-0-133-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-123.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-123.ec2.internal,},FirstTimestamp:2026-07-28 01:25:53.337352078 +0000 UTC m=+0.346870494,LastTimestamp:2026-07-28 01:25:53.337352078 +0000 UTC m=+0.346870494,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-123.ec2.internal,}"
Jul 28 01:25:53.339504 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.339492    2555 policy_none.go:49] "None policy: Start"
Jul 28 01:25:53.339621 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.339508    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 01:25:53.339621 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.339543    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 01:25:53.340613 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.340599    2555 policy_none.go:47] "Start"
Jul 28 01:25:53.342837 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.342821    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tkchw"
Jul 28 01:25:53.349104 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.349028    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-123.ec2.internal.18c64fb7d901e239  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-123.ec2.internal,UID:ip-10-0-133-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-123.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-123.ec2.internal,},FirstTimestamp:2026-07-28 01:25:53.337369145 +0000 UTC m=+0.346887561,LastTimestamp:2026-07-28 01:25:53.337369145 +0000 UTC m=+0.346887561,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-123.ec2.internal,}"
Jul 28 01:25:53.350542 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.350500    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tkchw"
Jul 28 01:25:53.372599 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.372584    2555 manager.go:337] "Starting Device Plugin manager"
Jul 28 01:25:53.372686 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.372618    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 01:25:53.372686 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.372631    2555 server.go:87] "Starting device plugin registration server"
Jul 28 01:25:53.372856 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.372844    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 01:25:53.372925 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.372858    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 01:25:53.372977 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.372928    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 01:25:53.373024 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.372993    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 01:25:53.373024 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.373001    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 01:25:53.373486 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.373469    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 01:25:53.373584 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.373509    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-123.ec2.internal\" not found"
Jul 28 01:25:53.445063 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.445036    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 01:25:53.446264 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.446245    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 01:25:53.446314 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.446296    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 01:25:53.446350 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.446320    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 01:25:53.446397 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.446359    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 01:25:53.448575 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.448554    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:53.473398 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.473372    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:53.474215 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.474201    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:53.474282 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.474226    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:53.474282 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.474237    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:53.474282 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.474263    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.481825 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.481810    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.481905 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.481832    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-123.ec2.internal\": node \"ip-10-0-133-123.ec2.internal\" not found"
Jul 28 01:25:53.498197 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.498150    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-123.ec2.internal\" not found"
Jul 28 01:25:53.547202 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.547184    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-123.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal"]
Jul 28 01:25:53.547266 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.547231    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:53.548515 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.548503    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:53.548607 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.548542    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:53.548607 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.548554    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:53.549714 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.549701    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:53.549834 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.549820    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.549886 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.549851    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:53.550339 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.550324    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:53.550339 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.550334    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:53.550448 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.550353    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:53.550448 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.550368    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:53.550448 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.550356    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:53.550448 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.550443    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:53.551790 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.551777    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.551836 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.551803    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:53.554412 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.554381    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:53.554501 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.554414    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:53.554501 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.554426    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:53.578060 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.578039    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-123.ec2.internal\" not found" node="ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.582747 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.582731    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-123.ec2.internal\" not found" node="ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.599062 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.599039    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-123.ec2.internal\" not found"
Jul 28 01:25:53.619503 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.619482    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3bb63b8ff0a229a53856e473495df153-config\") pod \"kube-apiserver-proxy-ip-10-0-133-123.ec2.internal\" (UID: \"3bb63b8ff0a229a53856e473495df153\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.619598 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.619505    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f5ac334478d72453f413c480d1005362-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal\" (UID: \"f5ac334478d72453f413c480d1005362\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.619598 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.619535    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5ac334478d72453f413c480d1005362-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal\" (UID: \"f5ac334478d72453f413c480d1005362\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.699482 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.699462    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-123.ec2.internal\" not found"
Jul 28 01:25:53.719740 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.719717    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5ac334478d72453f413c480d1005362-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal\" (UID: \"f5ac334478d72453f413c480d1005362\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.719818 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.719749    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3bb63b8ff0a229a53856e473495df153-config\") pod \"kube-apiserver-proxy-ip-10-0-133-123.ec2.internal\" (UID: \"3bb63b8ff0a229a53856e473495df153\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.719818 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.719774    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f5ac334478d72453f413c480d1005362-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal\" (UID: \"f5ac334478d72453f413c480d1005362\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.719917 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.719816    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5ac334478d72453f413c480d1005362-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal\" (UID: \"f5ac334478d72453f413c480d1005362\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.719917 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.719825    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f5ac334478d72453f413c480d1005362-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal\" (UID: \"f5ac334478d72453f413c480d1005362\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.719917 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.719829    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3bb63b8ff0a229a53856e473495df153-config\") pod \"kube-apiserver-proxy-ip-10-0-133-123.ec2.internal\" (UID: \"3bb63b8ff0a229a53856e473495df153\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.800107 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.800057    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-123.ec2.internal\" not found"
Jul 28 01:25:53.880572 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.880548    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.886182 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:53.886165    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal"
Jul 28 01:25:53.900153 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:53.900132    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-123.ec2.internal\" not found"
Jul 28 01:25:54.000620 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:54.000591    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-123.ec2.internal\" not found"
Jul 28 01:25:54.101145 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:54.101097    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-123.ec2.internal\" not found"
Jul 28 01:25:54.201715 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:54.201688    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-123.ec2.internal\" not found"
Jul 28 01:25:54.216503 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.216483    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:54.218461 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.218445    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal"
Jul 28 01:25:54.228771 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.228753    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 01:25:54.230411 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.230398    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-123.ec2.internal"
Jul 28 01:25:54.239454 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.239439    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 01:25:54.264557 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.264539    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 01:25:54.265013 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.264643    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 01:25:54.265013 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.264670    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 01:25:54.298852 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.298826    2555 apiserver.go:52] "Watching apiserver"
Jul 28 01:25:54.317175 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.317153    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 01:25:54.317509 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.317485    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-f4xdf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh","openshift-cluster-node-tuning-operator/tuned-pq94g","openshift-dns/node-resolver-k5qjk","openshift-image-registry/node-ca-z9b5j","openshift-multus/multus-9j9s8","openshift-multus/network-metrics-daemon-k24x4","openshift-network-operator/iptables-alerter-v7xpp","kube-system/konnectivity-agent-qpfgr","kube-system/kube-apiserver-proxy-ip-10-0-133-123.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal","openshift-multus/multus-additional-cni-plugins-x8rd2","openshift-network-diagnostics/network-check-target-q8gnk","openshift-ovn-kubernetes/ovnkube-node-54bw2"]
Jul 28 01:25:54.317608 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.317507    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 01:25:54.318896 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.318878    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:25:54.318992 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:54.318940    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:25:54.319932 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.319913    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.321245 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.321224    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.322289 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.322268    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 01:25:54.322383 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.322357    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 01:25:54.323042 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.322562    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d9zmp\""
Jul 28 01:25:54.323042 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.322695    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 01:25:54.323042 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.322952    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k5qjk"
Jul 28 01:25:54.323467 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.323421    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be56d98e-e52b-4ea8-9187-511176665495-dbus\") pod \"global-pull-secret-syncer-f4xdf\" (UID: \"be56d98e-e52b-4ea8-9187-511176665495\") " pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:25:54.323579 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.323483    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret\") pod \"global-pull-secret-syncer-f4xdf\" (UID: \"be56d98e-e52b-4ea8-9187-511176665495\") " pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:25:54.323579 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.323543    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be56d98e-e52b-4ea8-9187-511176665495-kubelet-config\") pod \"global-pull-secret-syncer-f4xdf\" (UID: \"be56d98e-e52b-4ea8-9187-511176665495\") " pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:25:54.324031 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.323801    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:25:54.324031 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.323884    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 01:25:54.324031 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.323944    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6g28l\""
Jul 28 01:25:54.325203 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.325187    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z9b5j"
Jul 28 01:25:54.325835 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.325815    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 01:25:54.325917 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.325880    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l98wn\""
Jul 28 01:25:54.325967 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.325941    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 01:25:54.326996 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.326975    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.327223 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.327203    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 01:25:54.328177 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.328160    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:25:54.328249 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:54.328214    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:25:54.328745 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.328730    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 01:25:54.328852 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.328771    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pr6f5\""
Jul 28 01:25:54.328963 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.328949    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 01:25:54.329308 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.329289    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 01:25:54.329396 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.329380    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v7xpp"
Jul 28 01:25:54.329827 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.329813    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 01:25:54.329899 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.329845    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6rbp2\""
Jul 28 01:25:54.329899 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.329846    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 01:25:54.329994 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.329848    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 01:25:54.329994 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.329848    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 01:25:54.330689 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.330673    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qpfgr"
Jul 28 01:25:54.331455 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.331442    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 01:25:54.331634 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.331618    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:25:54.331709 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.331636    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 01:25:54.331709 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.331647    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-24n92\""
Jul 28 01:25:54.331941 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.331928    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.332816 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.332801    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 01:25:54.332870 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.332834    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b7vs8\""
Jul 28 01:25:54.332870 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.332862    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 01:25:54.333179 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.333167    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:25:54.333237 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:54.333222    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:25:54.333927 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.333906    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 01:25:54.334198 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.334180    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 01:25:54.334261 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.334252    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zc97n\""
Jul 28 01:25:54.334559 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.334543    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.337074 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.337059    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 01:25:54.338174 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.338025    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 01:25:54.338174 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.338065    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 01:25:54.338174 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.338073    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 01:25:54.338174 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.338025    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 01:25:54.338335 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.338224    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 01:25:54.338459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.338444    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z97hf\""
Jul 28 01:25:54.352896 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.352859    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jdmxs"
Jul 28 01:25:54.355087 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.355057    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 01:20:54 +0000 UTC" deadline="2028-01-25 19:17:08.085123904 +0000 UTC"
Jul 28 01:25:54.355137 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.355088    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13121h51m13.730038321s"
Jul 28 01:25:54.358721 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.358703    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jdmxs"
Jul 28 01:25:54.393929 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:54.393904    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ac334478d72453f413c480d1005362.slice/crio-b3f6a1069ef2ef9fa13d13d1d357bcfbcbb50d5610dc5a3637e5ba79f44b5c1a WatchSource:0}: Error finding container b3f6a1069ef2ef9fa13d13d1d357bcfbcbb50d5610dc5a3637e5ba79f44b5c1a: Status 404 returned error can't find the container with id b3f6a1069ef2ef9fa13d13d1d357bcfbcbb50d5610dc5a3637e5ba79f44b5c1a
Jul 28 01:25:54.394185 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:54.394162    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb63b8ff0a229a53856e473495df153.slice/crio-9f7e11edc5b7b24fb6734899562924338e66dc54269723c3b68392f4b62893d7 WatchSource:0}: Error finding container 9f7e11edc5b7b24fb6734899562924338e66dc54269723c3b68392f4b62893d7: Status 404 returned error can't find the container with id 9f7e11edc5b7b24fb6734899562924338e66dc54269723c3b68392f4b62893d7
Jul 28 01:25:54.394890 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.394873    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:54.399215 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.399202    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:25:54.418716 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.418698    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 01:25:54.423918 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.423891    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-etc-selinux\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.423918 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.423916    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-run-netns\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.424033 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.423930    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-hostroot\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.424033 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.423944    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt4hg\" (UniqueName: \"kubernetes.io/projected/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-kube-api-access-dt4hg\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.424033 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.423958    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-kubelet\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.424033 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.423973    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-run-openvswitch\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.424033 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.423986    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9607f9f-591a-4033-83ec-44deaee179e6-ovn-node-metrics-cert\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.424033 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424027    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be56d98e-e52b-4ea8-9187-511176665495-dbus\") pod \"global-pull-secret-syncer-f4xdf\" (UID: \"be56d98e-e52b-4ea8-9187-511176665495\") " pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:25:54.424352 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424053    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/80b8c33c-c6e3-4540-ba14-4027f4147c7e-hosts-file\") pod \"node-resolver-k5qjk\" (UID: \"80b8c33c-c6e3-4540-ba14-4027f4147c7e\") " pod="openshift-dns/node-resolver-k5qjk"
Jul 28 01:25:54.424352 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424075    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ba62973-501c-45ad-9710-200326696aa3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.424352 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424099    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/965314c0-5f5a-4d0e-a133-9fd61ae4c83c-host\") pod \"node-ca-z9b5j\" (UID: \"965314c0-5f5a-4d0e-a133-9fd61ae4c83c\") " pod="openshift-image-registry/node-ca-z9b5j"
Jul 28 01:25:54.424352 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424118    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dff83d23-57b7-440c-b7a1-d2834e3d31c6-agent-certs\") pod \"konnectivity-agent-qpfgr\" (UID: \"dff83d23-57b7-440c-b7a1-d2834e3d31c6\") " pod="kube-system/konnectivity-agent-qpfgr"
Jul 28 01:25:54.424352 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424144    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be56d98e-e52b-4ea8-9187-511176665495-dbus\") pod \"global-pull-secret-syncer-f4xdf\" (UID: \"be56d98e-e52b-4ea8-9187-511176665495\") " pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:25:54.424352 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424147    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-kubernetes\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.424352 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424185    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xg6m\" (UniqueName: \"kubernetes.io/projected/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-kube-api-access-6xg6m\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.424352 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424208    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-log-socket\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.424352 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424229    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-cni-netd\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.424352 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424245    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-lib-modules\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.424352 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424262    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f547cf9-cb20-4d9b-8c20-1f66359d6103-host-slash\") pod \"iptables-alerter-v7xpp\" (UID: \"6f547cf9-cb20-4d9b-8c20-1f66359d6103\") " pod="openshift-network-operator/iptables-alerter-v7xpp"
Jul 28 01:25:54.424352 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424276    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ba62973-501c-45ad-9710-200326696aa3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.424352 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424289    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-var-lib-kubelet\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.424352 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424304    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmhg9\" (UniqueName: \"kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9\") pod \"network-check-target-q8gnk\" (UID: \"4ba73e34-6176-472c-bbde-ffd0c9b335eb\") " pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:25:54.424352 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424327    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.424352 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424346    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-socket-dir\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424360    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-device-dir\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424401    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpbgk\" (UniqueName: \"kubernetes.io/projected/965314c0-5f5a-4d0e-a133-9fd61ae4c83c-kube-api-access-cpbgk\") pod \"node-ca-z9b5j\" (UID: \"965314c0-5f5a-4d0e-a133-9fd61ae4c83c\") " pod="openshift-image-registry/node-ca-z9b5j"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424418    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs\") pod \"network-metrics-daemon-k24x4\" (UID: \"3442c3b8-9ee2-4ad0-bbcf-b0918932580e\") " pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424434    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dff83d23-57b7-440c-b7a1-d2834e3d31c6-konnectivity-ca\") pod \"konnectivity-agent-qpfgr\" (UID: \"dff83d23-57b7-440c-b7a1-d2834e3d31c6\") " pod="kube-system/konnectivity-agent-qpfgr"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424451    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-os-release\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424481    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9607f9f-591a-4033-83ec-44deaee179e6-env-overrides\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424501    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-run\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424532    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-tuned\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424553    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-cnibin\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424568    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjc5l\" (UniqueName: \"kubernetes.io/projected/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-kube-api-access-tjc5l\") pod \"network-metrics-daemon-k24x4\" (UID: \"3442c3b8-9ee2-4ad0-bbcf-b0918932580e\") " pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424582    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9607f9f-591a-4033-83ec-44deaee179e6-ovnkube-script-lib\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424598    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-var-lib-kubelet\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424621    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b8c33c-c6e3-4540-ba14-4027f4147c7e-tmp-dir\") pod \"node-resolver-k5qjk\" (UID: \"80b8c33c-c6e3-4540-ba14-4027f4147c7e\") " pod="openshift-dns/node-resolver-k5qjk"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424647    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-multus-socket-dir-parent\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424661    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ba62973-501c-45ad-9710-200326696aa3-system-cni-dir\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424700    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/965314c0-5f5a-4d0e-a133-9fd61ae4c83c-serviceca\") pod \"node-ca-z9b5j\" (UID: \"965314c0-5f5a-4d0e-a133-9fd61ae4c83c\") " pod="openshift-image-registry/node-ca-z9b5j"
Jul 28 01:25:54.425085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424722    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-run-systemd\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.425847 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424736    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-systemd\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.425847 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424771    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmfg7\" (UniqueName: \"kubernetes.io/projected/1e110244-7c32-4ee1-aa8a-7fc45babedac-kube-api-access-rmfg7\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.425847 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424793    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-var-lib-openvswitch\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.425847 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424813    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.425847 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424828    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-sysconfig\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.425847 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424847    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret\") pod \"global-pull-secret-syncer-f4xdf\" (UID: \"be56d98e-e52b-4ea8-9187-511176665495\") " pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:25:54.425847 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424865    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-system-cni-dir\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.425847 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424886    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-var-lib-cni-bin\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.425847 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424909    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-multus-conf-dir\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.425847 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424934    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-multus-daemon-config\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.425847 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:54.424940    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:54.425847 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.424957    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2ba62973-501c-45ad-9710-200326696aa3-os-release\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.425847 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:54.425006    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret podName:be56d98e-e52b-4ea8-9187-511176665495 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:54.924983526 +0000 UTC m=+1.934501928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret") pod "global-pull-secret-syncer-f4xdf" (UID: "be56d98e-e52b-4ea8-9187-511176665495") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:54.425847 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425043    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-modprobe-d\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.425847 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425061    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4b45\" (UniqueName: \"kubernetes.io/projected/80b8c33c-c6e3-4540-ba14-4027f4147c7e-kube-api-access-t4b45\") pod \"node-resolver-k5qjk\" (UID: \"80b8c33c-c6e3-4540-ba14-4027f4147c7e\") " pod="openshift-dns/node-resolver-k5qjk"
Jul 28 01:25:54.425847 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425079    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ffwt\" (UniqueName: \"kubernetes.io/projected/6f547cf9-cb20-4d9b-8c20-1f66359d6103-kube-api-access-5ffwt\") pod \"iptables-alerter-v7xpp\" (UID: \"6f547cf9-cb20-4d9b-8c20-1f66359d6103\") " pod="openshift-network-operator/iptables-alerter-v7xpp"
Jul 28 01:25:54.426654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425100    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-systemd-units\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.426654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425119    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-run-ovn\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.426654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425132    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-node-log\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.426654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425145    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-host\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.426654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425160    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-run-multus-certs\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.426654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425176    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ba62973-501c-45ad-9710-200326696aa3-cnibin\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.426654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425195    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ba62973-501c-45ad-9710-200326696aa3-cni-binary-copy\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.426654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425210    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66pxx\" (UniqueName: \"kubernetes.io/projected/b9607f9f-591a-4033-83ec-44deaee179e6-kube-api-access-66pxx\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.426654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425225    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-tmp\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.426654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425242    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be56d98e-e52b-4ea8-9187-511176665495-kubelet-config\") pod \"global-pull-secret-syncer-f4xdf\" (UID: \"be56d98e-e52b-4ea8-9187-511176665495\") " pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:25:54.426654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425256    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-multus-cni-dir\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.426654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425273    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6f547cf9-cb20-4d9b-8c20-1f66359d6103-iptables-alerter-script\") pod \"iptables-alerter-v7xpp\" (UID: \"6f547cf9-cb20-4d9b-8c20-1f66359d6103\") " pod="openshift-network-operator/iptables-alerter-v7xpp"
Jul 28 01:25:54.426654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425296    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwlbw\" (UniqueName: \"kubernetes.io/projected/2ba62973-501c-45ad-9710-200326696aa3-kube-api-access-hwlbw\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.426654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425314    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be56d98e-e52b-4ea8-9187-511176665495-kubelet-config\") pod \"global-pull-secret-syncer-f4xdf\" (UID: \"be56d98e-e52b-4ea8-9187-511176665495\") " pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:25:54.426654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425317    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-slash\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.426654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425343    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-etc-openvswitch\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.427447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425362    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.427447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425378    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-run-netns\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.427447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425391    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-var-lib-cni-multus\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.427447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425410    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-sys-fs\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.427447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425429    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-cni-bin\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.427447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425442    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-sysctl-conf\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.427447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425454    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-cni-binary-copy\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.427447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425470    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-etc-kubernetes\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.427447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425495    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-registration-dir\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.427447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425509    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9607f9f-591a-4033-83ec-44deaee179e6-ovnkube-config\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.427447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425541    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-sysctl-d\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.427447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425554    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-sys\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.427447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425577    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-run-k8s-cni-cncf-io\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.427447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.425594    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ba62973-501c-45ad-9710-200326696aa3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.449899 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.449849    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal" event={"ID":"f5ac334478d72453f413c480d1005362","Type":"ContainerStarted","Data":"b3f6a1069ef2ef9fa13d13d1d357bcfbcbb50d5610dc5a3637e5ba79f44b5c1a"}
Jul 28 01:25:54.450843 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.450823    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-123.ec2.internal" event={"ID":"3bb63b8ff0a229a53856e473495df153","Type":"ContainerStarted","Data":"9f7e11edc5b7b24fb6734899562924338e66dc54269723c3b68392f4b62893d7"}
Jul 28 01:25:54.525909 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.525882    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-system-cni-dir\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.525909 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.525907    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-var-lib-cni-bin\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.526116 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.525930    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-multus-conf-dir\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.526116 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.525945    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-multus-daemon-config\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.526116 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.525960    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2ba62973-501c-45ad-9710-200326696aa3-os-release\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.526116 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.525988    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-modprobe-d\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.526116 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526003    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-system-cni-dir\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.526116 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526003    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-var-lib-cni-bin\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.526116 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526032    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-multus-conf-dir\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.526116 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526073    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4b45\" (UniqueName: \"kubernetes.io/projected/80b8c33c-c6e3-4540-ba14-4027f4147c7e-kube-api-access-t4b45\") pod \"node-resolver-k5qjk\" (UID: \"80b8c33c-c6e3-4540-ba14-4027f4147c7e\") " pod="openshift-dns/node-resolver-k5qjk"
Jul 28 01:25:54.526116 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526088    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-modprobe-d\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.526116 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526091    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2ba62973-501c-45ad-9710-200326696aa3-os-release\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.526116 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526100    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffwt\" (UniqueName: \"kubernetes.io/projected/6f547cf9-cb20-4d9b-8c20-1f66359d6103-kube-api-access-5ffwt\") pod \"iptables-alerter-v7xpp\" (UID: \"6f547cf9-cb20-4d9b-8c20-1f66359d6103\") " pod="openshift-network-operator/iptables-alerter-v7xpp"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526133    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-systemd-units\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526162    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-run-ovn\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526186    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-node-log\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526208    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-host\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526228    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-run-ovn\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526234    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-run-multus-certs\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526229    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-systemd-units\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526265    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ba62973-501c-45ad-9710-200326696aa3-cnibin\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526270    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-node-log\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526273    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-run-multus-certs\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526293    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ba62973-501c-45ad-9710-200326696aa3-cni-binary-copy\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526308    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-host\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526328    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66pxx\" (UniqueName: \"kubernetes.io/projected/b9607f9f-591a-4033-83ec-44deaee179e6-kube-api-access-66pxx\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526351    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ba62973-501c-45ad-9710-200326696aa3-cnibin\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526351    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-tmp\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526470    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-multus-cni-dir\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526500    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6f547cf9-cb20-4d9b-8c20-1f66359d6103-iptables-alerter-script\") pod \"iptables-alerter-v7xpp\" (UID: \"6f547cf9-cb20-4d9b-8c20-1f66359d6103\") " pod="openshift-network-operator/iptables-alerter-v7xpp"
Jul 28 01:25:54.526633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526543    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwlbw\" (UniqueName: \"kubernetes.io/projected/2ba62973-501c-45ad-9710-200326696aa3-kube-api-access-hwlbw\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526570    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-slash\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526592    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-etc-openvswitch\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526601    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-multus-cni-dir\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526619    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-multus-daemon-config\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526616    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526657    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-slash\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526660    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-run-netns\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526913    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.526956    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-var-lib-cni-multus\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527104    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-etc-openvswitch\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527134    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ba62973-501c-45ad-9710-200326696aa3-cni-binary-copy\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527137    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-sys-fs\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527162    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-run-netns\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527185    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-cni-bin\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527193    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-sys-fs\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527257    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-sysctl-conf\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.527317 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527311    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-var-lib-cni-multus\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527353    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-cni-binary-copy\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527382    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-etc-kubernetes\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527389    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-sysctl-conf\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527438    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-etc-kubernetes\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527441    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-cni-bin\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527468    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-registration-dir\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527501    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9607f9f-591a-4033-83ec-44deaee179e6-ovnkube-config\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527552    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-sysctl-d\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527584    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-sys\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527502    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6f547cf9-cb20-4d9b-8c20-1f66359d6103-iptables-alerter-script\") pod \"iptables-alerter-v7xpp\" (UID: \"6f547cf9-cb20-4d9b-8c20-1f66359d6103\") " pod="openshift-network-operator/iptables-alerter-v7xpp"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527617    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-run-k8s-cni-cncf-io\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527644    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ba62973-501c-45ad-9710-200326696aa3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527677    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-etc-selinux\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527709    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-run-netns\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527740    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-hostroot\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527771    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt4hg\" (UniqueName: \"kubernetes.io/projected/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-kube-api-access-dt4hg\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.528120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527807    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-kubelet\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527848    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-run-openvswitch\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527880    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9607f9f-591a-4033-83ec-44deaee179e6-ovn-node-metrics-cert\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527915    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/80b8c33c-c6e3-4540-ba14-4027f4147c7e-hosts-file\") pod \"node-resolver-k5qjk\" (UID: \"80b8c33c-c6e3-4540-ba14-4027f4147c7e\") " pod="openshift-dns/node-resolver-k5qjk"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527928    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-cni-binary-copy\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527947    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ba62973-501c-45ad-9710-200326696aa3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527973    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/965314c0-5f5a-4d0e-a133-9fd61ae4c83c-host\") pod \"node-ca-z9b5j\" (UID: \"965314c0-5f5a-4d0e-a133-9fd61ae4c83c\") " pod="openshift-image-registry/node-ca-z9b5j"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528002    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-run-netns\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528006    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dff83d23-57b7-440c-b7a1-d2834e3d31c6-agent-certs\") pod \"konnectivity-agent-qpfgr\" (UID: \"dff83d23-57b7-440c-b7a1-d2834e3d31c6\") " pod="kube-system/konnectivity-agent-qpfgr"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528036    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-kubernetes\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528068    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xg6m\" (UniqueName: \"kubernetes.io/projected/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-kube-api-access-6xg6m\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528068    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-run-openvswitch\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528105    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9607f9f-591a-4033-83ec-44deaee179e6-ovnkube-config\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528124    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-sysctl-d\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528134    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-log-socket\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528108    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-hostroot\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528209    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-kubelet\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528345    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-kubernetes\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.528880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528373    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-sys\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527583    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-registration-dir\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528434    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-run-k8s-cni-cncf-io\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528568    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/965314c0-5f5a-4d0e-a133-9fd61ae4c83c-host\") pod \"node-ca-z9b5j\" (UID: \"965314c0-5f5a-4d0e-a133-9fd61ae4c83c\") " pod="openshift-image-registry/node-ca-z9b5j"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528619    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-log-socket\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528642    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ba62973-501c-45ad-9710-200326696aa3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528658    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-cni-netd\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528693    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-lib-modules\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528707    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/80b8c33c-c6e3-4540-ba14-4027f4147c7e-hosts-file\") pod \"node-resolver-k5qjk\" (UID: \"80b8c33c-c6e3-4540-ba14-4027f4147c7e\") " pod="openshift-dns/node-resolver-k5qjk"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528718    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f547cf9-cb20-4d9b-8c20-1f66359d6103-host-slash\") pod \"iptables-alerter-v7xpp\" (UID: \"6f547cf9-cb20-4d9b-8c20-1f66359d6103\") " pod="openshift-network-operator/iptables-alerter-v7xpp"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528756    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f547cf9-cb20-4d9b-8c20-1f66359d6103-host-slash\") pod \"iptables-alerter-v7xpp\" (UID: \"6f547cf9-cb20-4d9b-8c20-1f66359d6103\") " pod="openshift-network-operator/iptables-alerter-v7xpp"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528757    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ba62973-501c-45ad-9710-200326696aa3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528803    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-var-lib-kubelet\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528836    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhg9\" (UniqueName: \"kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9\") pod \"network-check-target-q8gnk\" (UID: \"4ba73e34-6176-472c-bbde-ffd0c9b335eb\") " pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528869    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528872    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ba62973-501c-45ad-9710-200326696aa3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528920    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-etc-selinux\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.529685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528937    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ba62973-501c-45ad-9710-200326696aa3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528943    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-host-var-lib-kubelet\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528945    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-lib-modules\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528951    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-socket-dir\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.528992    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-device-dir\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529001    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529028    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-socket-dir\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529028    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpbgk\" (UniqueName: \"kubernetes.io/projected/965314c0-5f5a-4d0e-a133-9fd61ae4c83c-kube-api-access-cpbgk\") pod \"node-ca-z9b5j\" (UID: \"965314c0-5f5a-4d0e-a133-9fd61ae4c83c\") " pod="openshift-image-registry/node-ca-z9b5j"
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529071    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs\") pod \"network-metrics-daemon-k24x4\" (UID: \"3442c3b8-9ee2-4ad0-bbcf-b0918932580e\") " pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529097    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dff83d23-57b7-440c-b7a1-d2834e3d31c6-konnectivity-ca\") pod \"konnectivity-agent-qpfgr\" (UID: \"dff83d23-57b7-440c-b7a1-d2834e3d31c6\") " pod="kube-system/konnectivity-agent-qpfgr"
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529123    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-os-release\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529153    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9607f9f-591a-4033-83ec-44deaee179e6-env-overrides\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529177    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-run\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529044    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-cni-netd\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529234    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e110244-7c32-4ee1-aa8a-7fc45babedac-device-dir\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:54.529266    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529290    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-run\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.530459 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:54.529327    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs podName:3442c3b8-9ee2-4ad0-bbcf-b0918932580e nodeName:}" failed. No retries permitted until 2026-07-28 01:25:55.029311585 +0000 UTC m=+2.038829987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs") pod "network-metrics-daemon-k24x4" (UID: "3442c3b8-9ee2-4ad0-bbcf-b0918932580e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529352    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-tuned\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529379    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-cnibin\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529406    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjc5l\" (UniqueName: \"kubernetes.io/projected/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-kube-api-access-tjc5l\") pod \"network-metrics-daemon-k24x4\" (UID: \"3442c3b8-9ee2-4ad0-bbcf-b0918932580e\") " pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529433    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9607f9f-591a-4033-83ec-44deaee179e6-ovnkube-script-lib\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529461    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-var-lib-kubelet\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529486    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b8c33c-c6e3-4540-ba14-4027f4147c7e-tmp-dir\") pod \"node-resolver-k5qjk\" (UID: \"80b8c33c-c6e3-4540-ba14-4027f4147c7e\") " pod="openshift-dns/node-resolver-k5qjk"
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529508    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-multus-socket-dir-parent\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529581    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-multus-socket-dir-parent\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529809    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dff83d23-57b7-440c-b7a1-d2834e3d31c6-konnectivity-ca\") pod \"konnectivity-agent-qpfgr\" (UID: \"dff83d23-57b7-440c-b7a1-d2834e3d31c6\") " pod="kube-system/konnectivity-agent-qpfgr"
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529906    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-cnibin\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529952    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ba62973-501c-45ad-9710-200326696aa3-system-cni-dir\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.529989    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/965314c0-5f5a-4d0e-a133-9fd61ae4c83c-serviceca\") pod \"node-ca-z9b5j\" (UID: \"965314c0-5f5a-4d0e-a133-9fd61ae4c83c\") " pod="openshift-image-registry/node-ca-z9b5j"
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530015    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9607f9f-591a-4033-83ec-44deaee179e6-ovnkube-script-lib\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530021    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-run-systemd\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530060    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-systemd\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530074    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-var-lib-kubelet\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.531039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530092    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmfg7\" (UniqueName: \"kubernetes.io/projected/1e110244-7c32-4ee1-aa8a-7fc45babedac-kube-api-access-rmfg7\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.531586 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530120    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-var-lib-openvswitch\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.531586 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530155    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.531586 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530186    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-sysconfig\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.531586 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530277    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-sysconfig\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.531586 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530321    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ba62973-501c-45ad-9710-200326696aa3-system-cni-dir\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.531586 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530364    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-systemd\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.531586 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530578    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-var-lib-openvswitch\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.531586 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530639    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.531586 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530280    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b8c33c-c6e3-4540-ba14-4027f4147c7e-tmp-dir\") pod \"node-resolver-k5qjk\" (UID: \"80b8c33c-c6e3-4540-ba14-4027f4147c7e\") " pod="openshift-dns/node-resolver-k5qjk"
Jul 28 01:25:54.531586 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530687    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/965314c0-5f5a-4d0e-a133-9fd61ae4c83c-serviceca\") pod \"node-ca-z9b5j\" (UID: \"965314c0-5f5a-4d0e-a133-9fd61ae4c83c\") " pod="openshift-image-registry/node-ca-z9b5j"
Jul 28 01:25:54.531586 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530742    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9607f9f-591a-4033-83ec-44deaee179e6-run-systemd\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.531586 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530778    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9607f9f-591a-4033-83ec-44deaee179e6-env-overrides\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.531586 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.530788    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-os-release\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.531586 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.527325    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 01:25:54.534421 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.534399    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-etc-tuned\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.534514 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.534450    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9607f9f-591a-4033-83ec-44deaee179e6-ovn-node-metrics-cert\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.534514 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.534456    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-tmp\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.534609 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.534574    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dff83d23-57b7-440c-b7a1-d2834e3d31c6-agent-certs\") pod \"konnectivity-agent-qpfgr\" (UID: \"dff83d23-57b7-440c-b7a1-d2834e3d31c6\") " pod="kube-system/konnectivity-agent-qpfgr"
Jul 28 01:25:54.537815 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.537793    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ffwt\" (UniqueName: \"kubernetes.io/projected/6f547cf9-cb20-4d9b-8c20-1f66359d6103-kube-api-access-5ffwt\") pod \"iptables-alerter-v7xpp\" (UID: \"6f547cf9-cb20-4d9b-8c20-1f66359d6103\") " pod="openshift-network-operator/iptables-alerter-v7xpp"
Jul 28 01:25:54.538153 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.538130    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66pxx\" (UniqueName: \"kubernetes.io/projected/b9607f9f-591a-4033-83ec-44deaee179e6-kube-api-access-66pxx\") pod \"ovnkube-node-54bw2\" (UID: \"b9607f9f-591a-4033-83ec-44deaee179e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.538219 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.538200    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4b45\" (UniqueName: \"kubernetes.io/projected/80b8c33c-c6e3-4540-ba14-4027f4147c7e-kube-api-access-t4b45\") pod \"node-resolver-k5qjk\" (UID: \"80b8c33c-c6e3-4540-ba14-4027f4147c7e\") " pod="openshift-dns/node-resolver-k5qjk"
Jul 28 01:25:54.538873 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:54.538853    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:54.538967 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:54.538881    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:54.538967 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:54.538895    2555 projected.go:196] Error preparing data for projected volume kube-api-access-hmhg9 for pod openshift-network-diagnostics/network-check-target-q8gnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:54.539046 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:54.538979    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9 podName:4ba73e34-6176-472c-bbde-ffd0c9b335eb nodeName:}" failed. No retries permitted until 2026-07-28 01:25:55.038944063 +0000 UTC m=+2.048462465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hmhg9" (UniqueName: "kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9") pod "network-check-target-q8gnk" (UID: "4ba73e34-6176-472c-bbde-ffd0c9b335eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:54.540728 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.540706    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xg6m\" (UniqueName: \"kubernetes.io/projected/82b589cd-bf47-4440-a8dd-5f5cbf9794cb-kube-api-access-6xg6m\") pod \"tuned-pq94g\" (UID: \"82b589cd-bf47-4440-a8dd-5f5cbf9794cb\") " pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.541065 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.541038    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmfg7\" (UniqueName: \"kubernetes.io/projected/1e110244-7c32-4ee1-aa8a-7fc45babedac-kube-api-access-rmfg7\") pod \"aws-ebs-csi-driver-node-9glfh\" (UID: \"1e110244-7c32-4ee1-aa8a-7fc45babedac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.541327 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.541312    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjc5l\" (UniqueName: \"kubernetes.io/projected/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-kube-api-access-tjc5l\") pod \"network-metrics-daemon-k24x4\" (UID: \"3442c3b8-9ee2-4ad0-bbcf-b0918932580e\") " pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:25:54.541973 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.541957    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpbgk\" (UniqueName: \"kubernetes.io/projected/965314c0-5f5a-4d0e-a133-9fd61ae4c83c-kube-api-access-cpbgk\") pod \"node-ca-z9b5j\" (UID: \"965314c0-5f5a-4d0e-a133-9fd61ae4c83c\") " pod="openshift-image-registry/node-ca-z9b5j"
Jul 28 01:25:54.542250 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.542235    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwlbw\" (UniqueName: \"kubernetes.io/projected/2ba62973-501c-45ad-9710-200326696aa3-kube-api-access-hwlbw\") pod \"multus-additional-cni-plugins-x8rd2\" (UID: \"2ba62973-501c-45ad-9710-200326696aa3\") " pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.542286 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.542249    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt4hg\" (UniqueName: \"kubernetes.io/projected/2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd-kube-api-access-dt4hg\") pod \"multus-9j9s8\" (UID: \"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd\") " pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.629130 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.629057    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:54.641007 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.640985    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh"
Jul 28 01:25:54.646217 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.646202    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pq94g"
Jul 28 01:25:54.649746 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:54.649723    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e110244_7c32_4ee1_aa8a_7fc45babedac.slice/crio-c5c301c3619e0a233f9e04d5a2fc5a102079f8af5a27d6dbb41fb3b7af35bdd1 WatchSource:0}: Error finding container c5c301c3619e0a233f9e04d5a2fc5a102079f8af5a27d6dbb41fb3b7af35bdd1: Status 404 returned error can't find the container with id c5c301c3619e0a233f9e04d5a2fc5a102079f8af5a27d6dbb41fb3b7af35bdd1
Jul 28 01:25:54.654882 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:54.654859    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b589cd_bf47_4440_a8dd_5f5cbf9794cb.slice/crio-b5697a2ae08d42b5f01be2e646ef2934f3fcdc5a37b65e6a0adf6186b64ba185 WatchSource:0}: Error finding container b5697a2ae08d42b5f01be2e646ef2934f3fcdc5a37b65e6a0adf6186b64ba185: Status 404 returned error can't find the container with id b5697a2ae08d42b5f01be2e646ef2934f3fcdc5a37b65e6a0adf6186b64ba185
Jul 28 01:25:54.678259 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.678239    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k5qjk"
Jul 28 01:25:54.687741 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:54.687716    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b8c33c_c6e3_4540_ba14_4027f4147c7e.slice/crio-994e4d8596e1d9069855aecaffc3888e643ca12e8909bcac21f4da4bcea9ba14 WatchSource:0}: Error finding container 994e4d8596e1d9069855aecaffc3888e643ca12e8909bcac21f4da4bcea9ba14: Status 404 returned error can't find the container with id 994e4d8596e1d9069855aecaffc3888e643ca12e8909bcac21f4da4bcea9ba14
Jul 28 01:25:54.700306 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.700288    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z9b5j"
Jul 28 01:25:54.706980 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:54.706960    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965314c0_5f5a_4d0e_a133_9fd61ae4c83c.slice/crio-484800d7f948af8df5551ee3cf0e81592babe120a492bba1789657951c632b5b WatchSource:0}: Error finding container 484800d7f948af8df5551ee3cf0e81592babe120a492bba1789657951c632b5b: Status 404 returned error can't find the container with id 484800d7f948af8df5551ee3cf0e81592babe120a492bba1789657951c632b5b
Jul 28 01:25:54.707761 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.707740    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9j9s8"
Jul 28 01:25:54.713306 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.713289    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v7xpp"
Jul 28 01:25:54.715058 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:54.715041    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd2db27_d5f8_4dab_a8dc_d75d4091a8fd.slice/crio-bc6d05d9caf560135e74948969a2888453068a19214ee8d8ed31eef0970ee062 WatchSource:0}: Error finding container bc6d05d9caf560135e74948969a2888453068a19214ee8d8ed31eef0970ee062: Status 404 returned error can't find the container with id bc6d05d9caf560135e74948969a2888453068a19214ee8d8ed31eef0970ee062
Jul 28 01:25:54.719123 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.719105    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qpfgr"
Jul 28 01:25:54.720663 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:54.720638    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f547cf9_cb20_4d9b_8c20_1f66359d6103.slice/crio-ff72efae9ae69110e6cbd72ae2f03de328ccc06364f180312de51670e3733ce5 WatchSource:0}: Error finding container ff72efae9ae69110e6cbd72ae2f03de328ccc06364f180312de51670e3733ce5: Status 404 returned error can't find the container with id ff72efae9ae69110e6cbd72ae2f03de328ccc06364f180312de51670e3733ce5
Jul 28 01:25:54.723500 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.723480    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x8rd2"
Jul 28 01:25:54.727135 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:54.727100    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff83d23_57b7_440c_b7a1_d2834e3d31c6.slice/crio-1df2e69e9a62f01dbc5f2f611a1888ec39271a1ec599c241fa2179f4a038588b WatchSource:0}: Error finding container 1df2e69e9a62f01dbc5f2f611a1888ec39271a1ec599c241fa2179f4a038588b: Status 404 returned error can't find the container with id 1df2e69e9a62f01dbc5f2f611a1888ec39271a1ec599c241fa2179f4a038588b
Jul 28 01:25:54.729079 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.729057    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:25:54.732699 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:54.732679    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba62973_501c_45ad_9710_200326696aa3.slice/crio-dc7542ead5b3e2110e129a28b1ed156c8a7f1577e6539560a056bd94b989a68f WatchSource:0}: Error finding container dc7542ead5b3e2110e129a28b1ed156c8a7f1577e6539560a056bd94b989a68f: Status 404 returned error can't find the container with id dc7542ead5b3e2110e129a28b1ed156c8a7f1577e6539560a056bd94b989a68f
Jul 28 01:25:54.737421 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:25:54.737403    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9607f9f_591a_4033_83ec_44deaee179e6.slice/crio-4115f2692f4873962253788613f2970d789042630bca3a1556b3a515df25f4c8 WatchSource:0}: Error finding container 4115f2692f4873962253788613f2970d789042630bca3a1556b3a515df25f4c8: Status 404 returned error can't find the container with id 4115f2692f4873962253788613f2970d789042630bca3a1556b3a515df25f4c8
Jul 28 01:25:54.933181 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:54.933097    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret\") pod \"global-pull-secret-syncer-f4xdf\" (UID: \"be56d98e-e52b-4ea8-9187-511176665495\") " pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:25:54.933328 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:54.933305    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:54.933384 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:54.933371    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret podName:be56d98e-e52b-4ea8-9187-511176665495 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:55.933350409 +0000 UTC m=+2.942868835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret") pod "global-pull-secret-syncer-f4xdf" (UID: "be56d98e-e52b-4ea8-9187-511176665495") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:55.034577 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.034526    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs\") pod \"network-metrics-daemon-k24x4\" (UID: \"3442c3b8-9ee2-4ad0-bbcf-b0918932580e\") " pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:25:55.034756 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:55.034658    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:55.034756 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:55.034738    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs podName:3442c3b8-9ee2-4ad0-bbcf-b0918932580e nodeName:}" failed. No retries permitted until 2026-07-28 01:25:56.034716631 +0000 UTC m=+3.044235040 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs") pod "network-metrics-daemon-k24x4" (UID: "3442c3b8-9ee2-4ad0-bbcf-b0918932580e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:55.135791 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.135755    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhg9\" (UniqueName: \"kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9\") pod \"network-check-target-q8gnk\" (UID: \"4ba73e34-6176-472c-bbde-ffd0c9b335eb\") " pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:25:55.135970 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:55.135924    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:55.135970 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:55.135946    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:55.135970 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:55.135959    2555 projected.go:196] Error preparing data for projected volume kube-api-access-hmhg9 for pod openshift-network-diagnostics/network-check-target-q8gnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:55.136146 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:55.136021    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9 podName:4ba73e34-6176-472c-bbde-ffd0c9b335eb nodeName:}" failed. No retries permitted until 2026-07-28 01:25:56.136001813 +0000 UTC m=+3.145520219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmhg9" (UniqueName: "kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9") pod "network-check-target-q8gnk" (UID: "4ba73e34-6176-472c-bbde-ffd0c9b335eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:55.267330 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.267123    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:55.360765 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.360686    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 01:20:55 +0000 UTC" deadline="2028-01-01 21:14:32.973821377 +0000 UTC"
Jul 28 01:25:55.360765 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.360761    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12547h48m37.613064959s"
Jul 28 01:25:55.449533 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.449498    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:25:55.449731 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:55.449662    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:25:55.450076 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.450053    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:25:55.450193 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:55.450171    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:25:55.471799 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.471703    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh" event={"ID":"1e110244-7c32-4ee1-aa8a-7fc45babedac","Type":"ContainerStarted","Data":"c5c301c3619e0a233f9e04d5a2fc5a102079f8af5a27d6dbb41fb3b7af35bdd1"}
Jul 28 01:25:55.495952 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.495841    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2" event={"ID":"b9607f9f-591a-4033-83ec-44deaee179e6","Type":"ContainerStarted","Data":"4115f2692f4873962253788613f2970d789042630bca3a1556b3a515df25f4c8"}
Jul 28 01:25:55.506073 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.506038    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8rd2" event={"ID":"2ba62973-501c-45ad-9710-200326696aa3","Type":"ContainerStarted","Data":"dc7542ead5b3e2110e129a28b1ed156c8a7f1577e6539560a056bd94b989a68f"}
Jul 28 01:25:55.528336 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.528262    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v7xpp" event={"ID":"6f547cf9-cb20-4d9b-8c20-1f66359d6103","Type":"ContainerStarted","Data":"ff72efae9ae69110e6cbd72ae2f03de328ccc06364f180312de51670e3733ce5"}
Jul 28 01:25:55.540373 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.540305    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9j9s8" event={"ID":"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd","Type":"ContainerStarted","Data":"bc6d05d9caf560135e74948969a2888453068a19214ee8d8ed31eef0970ee062"}
Jul 28 01:25:55.559231 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.559183    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qpfgr" event={"ID":"dff83d23-57b7-440c-b7a1-d2834e3d31c6","Type":"ContainerStarted","Data":"1df2e69e9a62f01dbc5f2f611a1888ec39271a1ec599c241fa2179f4a038588b"}
Jul 28 01:25:55.575155 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.575123    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z9b5j" event={"ID":"965314c0-5f5a-4d0e-a133-9fd61ae4c83c","Type":"ContainerStarted","Data":"484800d7f948af8df5551ee3cf0e81592babe120a492bba1789657951c632b5b"}
Jul 28 01:25:55.595809 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.595782    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k5qjk" event={"ID":"80b8c33c-c6e3-4540-ba14-4027f4147c7e","Type":"ContainerStarted","Data":"994e4d8596e1d9069855aecaffc3888e643ca12e8909bcac21f4da4bcea9ba14"}
Jul 28 01:25:55.609615 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.606420    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pq94g" event={"ID":"82b589cd-bf47-4440-a8dd-5f5cbf9794cb","Type":"ContainerStarted","Data":"b5697a2ae08d42b5f01be2e646ef2934f3fcdc5a37b65e6a0adf6186b64ba185"}
Jul 28 01:25:55.943182 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:55.943083    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret\") pod \"global-pull-secret-syncer-f4xdf\" (UID: \"be56d98e-e52b-4ea8-9187-511176665495\") " pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:25:55.943345 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:55.943240    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:55.943345 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:55.943307    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret podName:be56d98e-e52b-4ea8-9187-511176665495 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:57.943286736 +0000 UTC m=+4.952805144 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret") pod "global-pull-secret-syncer-f4xdf" (UID: "be56d98e-e52b-4ea8-9187-511176665495") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:56.043909 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:56.043840    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs\") pod \"network-metrics-daemon-k24x4\" (UID: \"3442c3b8-9ee2-4ad0-bbcf-b0918932580e\") " pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:25:56.044082 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:56.044034    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:56.044144 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:56.044100    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs podName:3442c3b8-9ee2-4ad0-bbcf-b0918932580e nodeName:}" failed. No retries permitted until 2026-07-28 01:25:58.044079855 +0000 UTC m=+5.053598263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs") pod "network-metrics-daemon-k24x4" (UID: "3442c3b8-9ee2-4ad0-bbcf-b0918932580e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:56.145628 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:56.145190    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhg9\" (UniqueName: \"kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9\") pod \"network-check-target-q8gnk\" (UID: \"4ba73e34-6176-472c-bbde-ffd0c9b335eb\") " pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:25:56.145628 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:56.145346    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:56.145628 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:56.145366    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:56.145628 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:56.145378    2555 projected.go:196] Error preparing data for projected volume kube-api-access-hmhg9 for pod openshift-network-diagnostics/network-check-target-q8gnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:56.145628 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:56.145437    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9 podName:4ba73e34-6176-472c-bbde-ffd0c9b335eb nodeName:}" failed. No retries permitted until 2026-07-28 01:25:58.145419049 +0000 UTC m=+5.154937473 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmhg9" (UniqueName: "kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9") pod "network-check-target-q8gnk" (UID: "4ba73e34-6176-472c-bbde-ffd0c9b335eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:56.231394 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:56.231246    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:56.361560 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:56.361292    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 01:20:55 +0000 UTC" deadline="2028-04-02 11:07:54.725150232 +0000 UTC"
Jul 28 01:25:56.361560 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:56.361334    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14745h41m58.363821069s"
Jul 28 01:25:56.447620 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:56.447364    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:25:56.447847 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:56.447727    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:25:57.448307 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:57.447743    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:25:57.448307 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:57.447918    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:25:57.448307 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:57.447955    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:25:57.448307 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:57.448084    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:25:57.963251 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:57.963208    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret\") pod \"global-pull-secret-syncer-f4xdf\" (UID: \"be56d98e-e52b-4ea8-9187-511176665495\") " pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:25:57.963454 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:57.963403    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:57.963519 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:57.963463    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret podName:be56d98e-e52b-4ea8-9187-511176665495 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:01.963446954 +0000 UTC m=+8.972965358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret") pod "global-pull-secret-syncer-f4xdf" (UID: "be56d98e-e52b-4ea8-9187-511176665495") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:58.063715 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:58.063647    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs\") pod \"network-metrics-daemon-k24x4\" (UID: \"3442c3b8-9ee2-4ad0-bbcf-b0918932580e\") " pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:25:58.063953 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:58.063841    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:58.063953 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:58.063908    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs podName:3442c3b8-9ee2-4ad0-bbcf-b0918932580e nodeName:}" failed. No retries permitted until 2026-07-28 01:26:02.063887633 +0000 UTC m=+9.073406038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs") pod "network-metrics-daemon-k24x4" (UID: "3442c3b8-9ee2-4ad0-bbcf-b0918932580e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:58.164699 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:58.164656    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhg9\" (UniqueName: \"kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9\") pod \"network-check-target-q8gnk\" (UID: \"4ba73e34-6176-472c-bbde-ffd0c9b335eb\") " pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:25:58.164882 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:58.164834    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:58.164882 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:58.164859    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:58.164882 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:58.164872    2555 projected.go:196] Error preparing data for projected volume kube-api-access-hmhg9 for pod openshift-network-diagnostics/network-check-target-q8gnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:58.165047 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:58.164932    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9 podName:4ba73e34-6176-472c-bbde-ffd0c9b335eb nodeName:}" failed. No retries permitted until 2026-07-28 01:26:02.164912883 +0000 UTC m=+9.174431292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmhg9" (UniqueName: "kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9") pod "network-check-target-q8gnk" (UID: "4ba73e34-6176-472c-bbde-ffd0c9b335eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:58.447623 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:58.447588    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:25:58.447807 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:58.447721    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:25:59.449992 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:59.449307    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:25:59.449992 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:59.449439    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:25:59.449992 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:25:59.449834    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:25:59.449992 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:25:59.449944    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:26:00.447290 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:00.447252    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:00.447498 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:00.447400    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:26:01.460567 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:01.460534    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:01.461585 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:01.461166    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:01.462511 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:01.462034    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:26:01.463408 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:01.463363    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:26:02.006208 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:02.005647    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret\") pod \"global-pull-secret-syncer-f4xdf\" (UID: \"be56d98e-e52b-4ea8-9187-511176665495\") " pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:02.006208 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:02.005797    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:02.006208 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:02.005851    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret podName:be56d98e-e52b-4ea8-9187-511176665495 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:10.005834315 +0000 UTC m=+17.015352719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret") pod "global-pull-secret-syncer-f4xdf" (UID: "be56d98e-e52b-4ea8-9187-511176665495") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:02.106413 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:02.106378    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs\") pod \"network-metrics-daemon-k24x4\" (UID: \"3442c3b8-9ee2-4ad0-bbcf-b0918932580e\") " pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:02.106618 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:02.106599    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:02.106683 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:02.106672    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs podName:3442c3b8-9ee2-4ad0-bbcf-b0918932580e nodeName:}" failed. No retries permitted until 2026-07-28 01:26:10.106653625 +0000 UTC m=+17.116172051 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs") pod "network-metrics-daemon-k24x4" (UID: "3442c3b8-9ee2-4ad0-bbcf-b0918932580e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:02.207685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:02.207643    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhg9\" (UniqueName: \"kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9\") pod \"network-check-target-q8gnk\" (UID: \"4ba73e34-6176-472c-bbde-ffd0c9b335eb\") " pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:02.207897 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:02.207878    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:02.207897 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:02.207898    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:02.208031 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:02.207911    2555 projected.go:196] Error preparing data for projected volume kube-api-access-hmhg9 for pod openshift-network-diagnostics/network-check-target-q8gnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:02.208031 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:02.207962    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9 podName:4ba73e34-6176-472c-bbde-ffd0c9b335eb nodeName:}" failed. No retries permitted until 2026-07-28 01:26:10.207948156 +0000 UTC m=+17.217466560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmhg9" (UniqueName: "kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9") pod "network-check-target-q8gnk" (UID: "4ba73e34-6176-472c-bbde-ffd0c9b335eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:02.447032 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:02.446960    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:02.447190 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:02.447080    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:26:03.447483 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:03.447447    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:03.447939 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:03.447554    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:26:03.447939 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:03.447643    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:03.447939 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:03.447732    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:26:04.446861 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:04.446830    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:04.447131 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:04.446964    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:26:05.449874 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:05.449842    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:05.450292 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:05.449972    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:26:05.450292 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:05.450042    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:05.450292 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:05.450156    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:26:06.446824 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:06.446796    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:06.446960 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:06.446892    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:26:07.447039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:07.446992    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:07.447039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:07.447041    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:07.447558 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:07.447133    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:26:07.447558 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:07.447219    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:26:08.446686 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:08.446657    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:08.446854 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:08.446755    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:26:09.446638 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:09.446605    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:09.447115 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:09.446606    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:09.447115 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:09.446735    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:26:09.447115 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:09.446809    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:26:10.065887 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:10.065854    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret\") pod \"global-pull-secret-syncer-f4xdf\" (UID: \"be56d98e-e52b-4ea8-9187-511176665495\") " pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:10.066072 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:10.065976    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:10.066072 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:10.066028    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret podName:be56d98e-e52b-4ea8-9187-511176665495 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:26.066014341 +0000 UTC m=+33.075532759 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret") pod "global-pull-secret-syncer-f4xdf" (UID: "be56d98e-e52b-4ea8-9187-511176665495") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:10.166797 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:10.166769    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs\") pod \"network-metrics-daemon-k24x4\" (UID: \"3442c3b8-9ee2-4ad0-bbcf-b0918932580e\") " pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:10.166967 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:10.166936    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:10.167016 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:10.166992    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs podName:3442c3b8-9ee2-4ad0-bbcf-b0918932580e nodeName:}" failed. No retries permitted until 2026-07-28 01:26:26.166977124 +0000 UTC m=+33.176495526 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs") pod "network-metrics-daemon-k24x4" (UID: "3442c3b8-9ee2-4ad0-bbcf-b0918932580e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:10.267385 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:10.267357    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhg9\" (UniqueName: \"kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9\") pod \"network-check-target-q8gnk\" (UID: \"4ba73e34-6176-472c-bbde-ffd0c9b335eb\") " pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:10.267634 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:10.267540    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:10.267634 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:10.267564    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:10.267634 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:10.267574    2555 projected.go:196] Error preparing data for projected volume kube-api-access-hmhg9 for pod openshift-network-diagnostics/network-check-target-q8gnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:10.267634 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:10.267620    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9 podName:4ba73e34-6176-472c-bbde-ffd0c9b335eb nodeName:}" failed. No retries permitted until 2026-07-28 01:26:26.26760538 +0000 UTC m=+33.277123786 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmhg9" (UniqueName: "kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9") pod "network-check-target-q8gnk" (UID: "4ba73e34-6176-472c-bbde-ffd0c9b335eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:10.447604 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:10.447517    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:10.448099 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:10.447645    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:26:11.446648 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:11.446613    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:11.446832 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:11.446621    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:11.446832 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:11.446733    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:26:11.446938 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:11.446829    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:26:12.446721 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:12.446679    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:12.447115 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:12.446819    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:26:13.448087 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:13.448055    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:13.448544 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:13.448164    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:26:13.448544 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:13.448220    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:13.448544 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:13.448295    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:26:14.446655 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:14.446618    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:14.446824 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:14.446721    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:26:15.447283 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.446989    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:15.448137 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.447054    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:15.448137 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:15.447362    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:26:15.448137 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:15.447439    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:26:15.657993 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.657924    2555 generic.go:356] "Generic (PLEG): container finished" podID="f5ac334478d72453f413c480d1005362" containerID="d82125a8e0b6d792c2af9ce309719d272d86878cc622f79363f652dc91a36578" exitCode=0
Jul 28 01:26:15.658224 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.657996    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal" event={"ID":"f5ac334478d72453f413c480d1005362","Type":"ContainerDied","Data":"d82125a8e0b6d792c2af9ce309719d272d86878cc622f79363f652dc91a36578"}
Jul 28 01:26:15.659359 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.659323    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-123.ec2.internal" event={"ID":"3bb63b8ff0a229a53856e473495df153","Type":"ContainerStarted","Data":"303a69ff25570601b56a38af8c92bf5b7dc5291b6b93bf5ec1e482667a9aa1b5"}
Jul 28 01:26:15.660510 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.660490    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qpfgr" event={"ID":"dff83d23-57b7-440c-b7a1-d2834e3d31c6","Type":"ContainerStarted","Data":"7e1ffcb3c8b15d1a9e168cf7226c9622efbe43e6858eda4f042cdb735a2309bd"}
Jul 28 01:26:15.661780 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.661755    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z9b5j" event={"ID":"965314c0-5f5a-4d0e-a133-9fd61ae4c83c","Type":"ContainerStarted","Data":"d8f928c0ef8afede8c72b52586664ae9202527af92e59e53e7b4c4c304ed6189"}
Jul 28 01:26:15.662964 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.662945    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k5qjk" event={"ID":"80b8c33c-c6e3-4540-ba14-4027f4147c7e","Type":"ContainerStarted","Data":"65be7d2fc4af009f58560baf762e67bd067dd569c465527b59f4e8f2243ff6f8"}
Jul 28 01:26:15.664480 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.664457    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pq94g" event={"ID":"82b589cd-bf47-4440-a8dd-5f5cbf9794cb","Type":"ContainerStarted","Data":"5b3eedc6a1dba60c96a344197eaef07cce157426f175bcc32b5f9acc47792cc4"}
Jul 28 01:26:15.665973 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.665954    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh" event={"ID":"1e110244-7c32-4ee1-aa8a-7fc45babedac","Type":"ContainerStarted","Data":"025c8beb8c5f597a878db9fcb78297483c2e36e74857ac6880c3f4f6f1a4bfb2"}
Jul 28 01:26:15.669280 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.669256    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2" event={"ID":"b9607f9f-591a-4033-83ec-44deaee179e6","Type":"ContainerStarted","Data":"2a122450f0dce0bfd663aa590489aa6e00d658bc2a2a65b6fc5c5e2833b4d33d"}
Jul 28 01:26:15.669377 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.669329    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2" event={"ID":"b9607f9f-591a-4033-83ec-44deaee179e6","Type":"ContainerStarted","Data":"98dd30901f87991c8c2fb34be6913f77f7578acd6ce64356379ecb6728b7559d"}
Jul 28 01:26:15.669377 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.669368    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2" event={"ID":"b9607f9f-591a-4033-83ec-44deaee179e6","Type":"ContainerStarted","Data":"67ca86cbe8a35a7c6065e65b1d816d55fcffa19a3ebefad58236e28688b76a9b"}
Jul 28 01:26:15.669456 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.669423    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2" event={"ID":"b9607f9f-591a-4033-83ec-44deaee179e6","Type":"ContainerStarted","Data":"dff90904216e3d67688fd1477abe9644fedfa974adbad1cd2eb06c36c34c2db5"}
Jul 28 01:26:15.669494 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.669468    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2" event={"ID":"b9607f9f-591a-4033-83ec-44deaee179e6","Type":"ContainerStarted","Data":"e6d5dfaae7c20cb21159df2678a13bfa8f81ce1e52fd1768d777376515c6df22"}
Jul 28 01:26:15.669551 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.669534    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2" event={"ID":"b9607f9f-591a-4033-83ec-44deaee179e6","Type":"ContainerStarted","Data":"2a9b43bfefd61c184873659d9461065426064b054f4d82c22c79ef3cc1a7d8df"}
Jul 28 01:26:15.670899 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.670873    2555 generic.go:356] "Generic (PLEG): container finished" podID="2ba62973-501c-45ad-9710-200326696aa3" containerID="f545fa2933b7537d676d197bec739e9b6d32df084f0e9493f35f9ac20e3e9db7" exitCode=0
Jul 28 01:26:15.671003 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.670949    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8rd2" event={"ID":"2ba62973-501c-45ad-9710-200326696aa3","Type":"ContainerDied","Data":"f545fa2933b7537d676d197bec739e9b6d32df084f0e9493f35f9ac20e3e9db7"}
Jul 28 01:26:15.672581 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.672536    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-123.ec2.internal" podStartSLOduration=21.672504304 podStartE2EDuration="21.672504304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:15.671480987 +0000 UTC m=+22.680999439" watchObservedRunningTime="2026-07-28 01:26:15.672504304 +0000 UTC m=+22.682022729"
Jul 28 01:26:15.672881 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.672860    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9j9s8" event={"ID":"2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd","Type":"ContainerStarted","Data":"beeacb1cb47342b68768b3435856bda0ea79525808ba69d30ef51a8d9e43fedd"}
Jul 28 01:26:15.686608 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.686488    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pq94g" podStartSLOduration=1.586280425 podStartE2EDuration="21.686477246s" totalImagesPullingTime="20.100196821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:15.685946592 +0000 UTC m=+22.695465015" watchObservedRunningTime="2026-07-28 01:26:15.686477246 +0000 UTC m=+22.695995667"
Jul 28 01:26:15.714490 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.714443    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-k5qjk" podStartSLOduration=1.649395927 podStartE2EDuration="21.714426121s" totalImagesPullingTime="20.065030194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:15.701609077 +0000 UTC m=+22.711127502" watchObservedRunningTime="2026-07-28 01:26:15.714426121 +0000 UTC m=+22.723944535"
Jul 28 01:26:15.714612 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.714572    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z9b5j" podStartSLOduration=1.668890733 podStartE2EDuration="21.714561804s" totalImagesPullingTime="20.045671071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:15.714553296 +0000 UTC m=+22.724071710" watchObservedRunningTime="2026-07-28 01:26:15.714561804 +0000 UTC m=+22.724080230"
Jul 28 01:26:15.747080 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.747042    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qpfgr" podStartSLOduration=1.7241465059999999 podStartE2EDuration="21.747030459s" totalImagesPullingTime="20.022883953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:15.746380754 +0000 UTC m=+22.755899196" watchObservedRunningTime="2026-07-28 01:26:15.747030459 +0000 UTC m=+22.756548883"
Jul 28 01:26:15.761112 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:15.761072    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9j9s8" podStartSLOduration=1.660862429 podStartE2EDuration="21.761056854s" totalImagesPullingTime="20.100194425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:15.760708151 +0000 UTC m=+22.770226578" watchObservedRunningTime="2026-07-28 01:26:15.761056854 +0000 UTC m=+22.770575279"
Jul 28 01:26:16.391262 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:16.391236    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 01:26:16.391402 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:16.391302    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T01:26:16.391258242Z","UUID":"07aeceb1-87b8-429a-a20d-ef916d8b53a2","Handler":null,"Name":"","Endpoint":""}
Jul 28 01:26:16.392620 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:16.392602    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 01:26:16.392703 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:16.392641    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 01:26:16.447285 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:16.447258    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:16.447438 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:16.447356    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:26:16.677394 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:16.677318    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh" event={"ID":"1e110244-7c32-4ee1-aa8a-7fc45babedac","Type":"ContainerStarted","Data":"b882fcdae9466046e937d79f5e654de1485c1fb216d799609ad1c23f56097398"}
Jul 28 01:26:16.678812 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:16.678779    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v7xpp" event={"ID":"6f547cf9-cb20-4d9b-8c20-1f66359d6103","Type":"ContainerStarted","Data":"d80993690a617645164cee044394c86f0387e32ac07bb47b1663c607e0236ba2"}
Jul 28 01:26:16.693022 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:16.692980    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v7xpp" podStartSLOduration=2.662425506 podStartE2EDuration="22.692962296s" totalImagesPullingTime="20.03053679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:16.692595277 +0000 UTC m=+23.702113701" watchObservedRunningTime="2026-07-28 01:26:16.692962296 +0000 UTC m=+23.702480721"
Jul 28 01:26:17.447113 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:17.447084    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:17.447242 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:17.447218    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:26:17.447563 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:17.447517    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:17.447915 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:17.447657    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:26:17.685113 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:17.685036    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal" event={"ID":"f5ac334478d72453f413c480d1005362","Type":"ContainerStarted","Data":"9ef9fd58d392f1fa5e6d434b02ea9a5870c549dbe620d158d69a392e407d1805"}
Jul 28 01:26:17.687973 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:17.687553    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh" event={"ID":"1e110244-7c32-4ee1-aa8a-7fc45babedac","Type":"ContainerStarted","Data":"a823fed1e81f410dd0fdd8912c50923326589ced125a48a3b9c9764fbb301076"}
Jul 28 01:26:17.691890 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:17.691862    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2" event={"ID":"b9607f9f-591a-4033-83ec-44deaee179e6","Type":"ContainerStarted","Data":"3f68ec8e3dd6043dc15cbea27033774e413cc710a1f1e7cb22fbefc4b5f80b2a"}
Jul 28 01:26:17.699221 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:17.699169    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-123.ec2.internal" podStartSLOduration=23.699153672 podStartE2EDuration="23.699153672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:17.698661863 +0000 UTC m=+24.708180291" watchObservedRunningTime="2026-07-28 01:26:17.699153672 +0000 UTC m=+24.708672092"
Jul 28 01:26:17.713652 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:17.713610    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glfh" podStartSLOduration=1.567255826 podStartE2EDuration="23.713596675s" totalImagesPullingTime="22.146340849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:17.713370685 +0000 UTC m=+24.722889111" watchObservedRunningTime="2026-07-28 01:26:17.713596675 +0000 UTC m=+24.723115099"
Jul 28 01:26:18.446895 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:18.446856    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:18.447074 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:18.446959    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:26:19.447056 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:19.446898    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:19.447446 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:19.447131    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:26:19.447446 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:19.446898    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:19.447446 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:19.447246    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:26:19.807911 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:19.807814    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qpfgr"
Jul 28 01:26:19.809199 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:19.808447    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qpfgr"
Jul 28 01:26:20.446551 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:20.446469    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:20.446839 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:20.446577    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:26:20.704087 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:20.704012    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2" event={"ID":"b9607f9f-591a-4033-83ec-44deaee179e6","Type":"ContainerStarted","Data":"4bdb5ba51a82f15f1e535a61ead6e7522526e42da83a4468f7b6db672f0385c1"}
Jul 28 01:26:20.704878 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:20.704247    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:26:20.706025 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:20.706003    2555 generic.go:356] "Generic (PLEG): container finished" podID="2ba62973-501c-45ad-9710-200326696aa3" containerID="cc3cdda30382ee9a2d98fe1b7577873323e46d46d0c17903eeef4f1f7840dd38" exitCode=0
Jul 28 01:26:20.706146 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:20.706088    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8rd2" event={"ID":"2ba62973-501c-45ad-9710-200326696aa3","Type":"ContainerDied","Data":"cc3cdda30382ee9a2d98fe1b7577873323e46d46d0c17903eeef4f1f7840dd38"}
Jul 28 01:26:20.706189 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:20.706163    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qpfgr"
Jul 28 01:26:20.707220 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:20.706697    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qpfgr"
Jul 28 01:26:20.720926 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:20.720770    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:26:20.730094 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:20.730062    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2" podStartSLOduration=6.653832755 podStartE2EDuration="26.730050703s" totalImagesPullingTime="20.076217948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:20.729944022 +0000 UTC m=+27.739462447" watchObservedRunningTime="2026-07-28 01:26:20.730050703 +0000 UTC m=+27.739569128"
Jul 28 01:26:21.447474 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:21.447446    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:21.447746 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:21.447475    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:21.447746 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:21.447622    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:26:21.447869 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:21.447805    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:26:21.708116 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:21.708091    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:26:21.708474 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:21.708174    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:26:21.723479 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:21.723459    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:26:21.937500 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:21.937467    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f4xdf"]
Jul 28 01:26:21.937668 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:21.937654    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:21.937847 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:21.937780    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:26:21.937968 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:21.937844    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q8gnk"]
Jul 28 01:26:21.937968 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:21.937955    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:21.938088 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:21.938049    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:26:21.938404 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:21.938380    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k24x4"]
Jul 28 01:26:21.938482 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:21.938470    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:21.938606 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:21.938585    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:26:22.712466 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:22.712431    2555 generic.go:356] "Generic (PLEG): container finished" podID="2ba62973-501c-45ad-9710-200326696aa3" containerID="77ec567173857392bac0e632ea7be8ee670fa9e2423d35ebb2b032e1674f51ca" exitCode=0
Jul 28 01:26:22.713069 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:22.712542    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8rd2" event={"ID":"2ba62973-501c-45ad-9710-200326696aa3","Type":"ContainerDied","Data":"77ec567173857392bac0e632ea7be8ee670fa9e2423d35ebb2b032e1674f51ca"}
Jul 28 01:26:23.448088 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:23.448061    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:23.448238 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:23.448144    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:26:23.448238 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:23.448228    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:23.448358 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:23.448335    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:26:23.448401 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:23.448391    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:23.448490 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:23.448475    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:26:24.719778 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:24.719621    2555 generic.go:356] "Generic (PLEG): container finished" podID="2ba62973-501c-45ad-9710-200326696aa3" containerID="9ea1103951ebd4588f82e8109acbde3dad3b7f5fe56d930d098dd1a7327019bc" exitCode=0
Jul 28 01:26:24.719778 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:24.719700    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8rd2" event={"ID":"2ba62973-501c-45ad-9710-200326696aa3","Type":"ContainerDied","Data":"9ea1103951ebd4588f82e8109acbde3dad3b7f5fe56d930d098dd1a7327019bc"}
Jul 28 01:26:25.447605 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:25.447573    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:25.447787 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:25.447580    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:25.447787 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:25.447729    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:25.447787 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:25.447686    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8gnk" podUID="4ba73e34-6176-472c-bbde-ffd0c9b335eb"
Jul 28 01:26:25.447940 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:25.447789    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k24x4" podUID="3442c3b8-9ee2-4ad0-bbcf-b0918932580e"
Jul 28 01:26:25.447940 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:25.447852    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xdf" podUID="be56d98e-e52b-4ea8-9187-511176665495"
Jul 28 01:26:26.090366 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:26.090332    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret\") pod \"global-pull-secret-syncer-f4xdf\" (UID: \"be56d98e-e52b-4ea8-9187-511176665495\") " pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:26.090836 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:26.090491    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:26.090836 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:26.090580    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret podName:be56d98e-e52b-4ea8-9187-511176665495 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:58.090559258 +0000 UTC m=+65.100077667 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret") pod "global-pull-secret-syncer-f4xdf" (UID: "be56d98e-e52b-4ea8-9187-511176665495") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:26.190682 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:26.190653    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs\") pod \"network-metrics-daemon-k24x4\" (UID: \"3442c3b8-9ee2-4ad0-bbcf-b0918932580e\") " pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:26.190841 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:26.190780    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:26.190841 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:26.190834    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs podName:3442c3b8-9ee2-4ad0-bbcf-b0918932580e nodeName:}" failed. No retries permitted until 2026-07-28 01:26:58.190816988 +0000 UTC m=+65.200335396 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs") pod "network-metrics-daemon-k24x4" (UID: "3442c3b8-9ee2-4ad0-bbcf-b0918932580e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:26.291763 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:26.291721    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhg9\" (UniqueName: \"kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9\") pod \"network-check-target-q8gnk\" (UID: \"4ba73e34-6176-472c-bbde-ffd0c9b335eb\") " pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:26.291983 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:26.291858    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:26.291983 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:26.291877    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:26.291983 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:26.291891    2555 projected.go:196] Error preparing data for projected volume kube-api-access-hmhg9 for pod openshift-network-diagnostics/network-check-target-q8gnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:26.291983 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:26.291948    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9 podName:4ba73e34-6176-472c-bbde-ffd0c9b335eb nodeName:}" failed. No retries permitted until 2026-07-28 01:26:58.291933117 +0000 UTC m=+65.301451521 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmhg9" (UniqueName: "kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9") pod "network-check-target-q8gnk" (UID: "4ba73e34-6176-472c-bbde-ffd0c9b335eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:27.316314 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.316289    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-123.ec2.internal" event="NodeReady"
Jul 28 01:26:27.316777 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.316445    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 01:26:27.351968 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.351182    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"]
Jul 28 01:26:27.372711 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.372689    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v6xwj"]
Jul 28 01:26:27.372844 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.372810    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.375613 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.375399    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 01:26:27.375613 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.375403    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tgrjr\""
Jul 28 01:26:27.375613 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.375503    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 01:26:27.375613 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.375597    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 01:26:27.381915 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.381896    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 01:26:27.382719 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.382701    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7kw4c"]
Jul 28 01:26:27.382854 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.382831    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:27.385210 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.385192    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 01:26:27.385296 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.385266    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wtm4s\""
Jul 28 01:26:27.385334 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.385194    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 01:26:27.388854 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.388836    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"]
Jul 28 01:26:27.388942 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.388867    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7kw4c"]
Jul 28 01:26:27.388942 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.388885    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v6xwj"]
Jul 28 01:26:27.389019 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.388982    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7kw4c"
Jul 28 01:26:27.391390 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.391374    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 01:26:27.391480 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.391390    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 01:26:27.391480 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.391450    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t6tn5\""
Jul 28 01:26:27.391684 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.391669    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 01:26:27.447356 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.447326    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:27.447356 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.447342    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:27.447603 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.447328    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:27.450688 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.450556    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 01:26:27.450688 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.450578    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 01:26:27.450688 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.450587    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 01:26:27.450688 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.450564    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6xpqm\""
Jul 28 01:26:27.450688 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.450619    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ptgdn\""
Jul 28 01:26:27.450688 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.450593    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 01:26:27.501088 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.501046    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa0f7d7b-c27c-476f-a48c-6a27b3057102-tmp-dir\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:27.501222 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.501118    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-certificates\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.501222 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.501147    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a37d3e7a-a5f2-465b-b814-31c0dc43842f-cert\") pod \"ingress-canary-7kw4c\" (UID: \"a37d3e7a-a5f2-465b-b814-31c0dc43842f\") " pod="openshift-ingress-canary/ingress-canary-7kw4c"
Jul 28 01:26:27.501222 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.501171    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcefd3df-d067-4b6c-a53e-216c8afb3788-installation-pull-secrets\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.501222 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.501196    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-bound-sa-token\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.501413 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.501223    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcefd3df-d067-4b6c-a53e-216c8afb3788-image-registry-private-configuration\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.501413 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.501271    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbrtd\" (UniqueName: \"kubernetes.io/projected/a37d3e7a-a5f2-465b-b814-31c0dc43842f-kube-api-access-sbrtd\") pod \"ingress-canary-7kw4c\" (UID: \"a37d3e7a-a5f2-465b-b814-31c0dc43842f\") " pod="openshift-ingress-canary/ingress-canary-7kw4c"
Jul 28 01:26:27.501413 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.501318    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:27.501413 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.501379    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.501597 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.501411    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcefd3df-d067-4b6c-a53e-216c8afb3788-trusted-ca\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.501597 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.501466    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r7h8\" (UniqueName: \"kubernetes.io/projected/aa0f7d7b-c27c-476f-a48c-6a27b3057102-kube-api-access-4r7h8\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:27.501597 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.501494    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78flt\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-kube-api-access-78flt\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.501597 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.501516    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcefd3df-d067-4b6c-a53e-216c8afb3788-ca-trust-extracted\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.501597 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.501558    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa0f7d7b-c27c-476f-a48c-6a27b3057102-config-volume\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:27.602464 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.602401    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa0f7d7b-c27c-476f-a48c-6a27b3057102-tmp-dir\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:27.602464 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.602456    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-certificates\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.602693 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.602670    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a37d3e7a-a5f2-465b-b814-31c0dc43842f-cert\") pod \"ingress-canary-7kw4c\" (UID: \"a37d3e7a-a5f2-465b-b814-31c0dc43842f\") " pod="openshift-ingress-canary/ingress-canary-7kw4c"
Jul 28 01:26:27.602748 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.602706    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcefd3df-d067-4b6c-a53e-216c8afb3788-installation-pull-secrets\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.602748 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.602725    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-bound-sa-token\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.602748 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.602744    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcefd3df-d067-4b6c-a53e-216c8afb3788-image-registry-private-configuration\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.602868 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.602795    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbrtd\" (UniqueName: \"kubernetes.io/projected/a37d3e7a-a5f2-465b-b814-31c0dc43842f-kube-api-access-sbrtd\") pod \"ingress-canary-7kw4c\" (UID: \"a37d3e7a-a5f2-465b-b814-31c0dc43842f\") " pod="openshift-ingress-canary/ingress-canary-7kw4c"
Jul 28 01:26:27.602868 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.602800    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa0f7d7b-c27c-476f-a48c-6a27b3057102-tmp-dir\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:27.602868 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.602817    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:27.602868 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.602862    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.603013 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.602890    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcefd3df-d067-4b6c-a53e-216c8afb3788-trusted-ca\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.603013 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:27.602898    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:27.603013 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.602920    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r7h8\" (UniqueName: \"kubernetes.io/projected/aa0f7d7b-c27c-476f-a48c-6a27b3057102-kube-api-access-4r7h8\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:27.603013 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:27.602941    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls podName:aa0f7d7b-c27c-476f-a48c-6a27b3057102 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:28.102929188 +0000 UTC m=+35.112447590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls") pod "dns-default-v6xwj" (UID: "aa0f7d7b-c27c-476f-a48c-6a27b3057102") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:27.603013 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.602962    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78flt\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-kube-api-access-78flt\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.603013 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.602982    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcefd3df-d067-4b6c-a53e-216c8afb3788-ca-trust-extracted\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.603013 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.603004    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa0f7d7b-c27c-476f-a48c-6a27b3057102-config-volume\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:27.603013 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.603005    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-certificates\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.603351 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:27.603338    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 01:26:27.603390 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:27.603355    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8ff4cf8b7-vw2rp: secret "image-registry-tls" not found
Jul 28 01:26:27.603418 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:27.603400    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls podName:fcefd3df-d067-4b6c-a53e-216c8afb3788 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:28.103385502 +0000 UTC m=+35.112903908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls") pod "image-registry-8ff4cf8b7-vw2rp" (UID: "fcefd3df-d067-4b6c-a53e-216c8afb3788") : secret "image-registry-tls" not found
Jul 28 01:26:27.603717 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.603691    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcefd3df-d067-4b6c-a53e-216c8afb3788-ca-trust-extracted\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.603997 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.603979    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa0f7d7b-c27c-476f-a48c-6a27b3057102-config-volume\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:27.604699 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.604679    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcefd3df-d067-4b6c-a53e-216c8afb3788-trusted-ca\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.607984 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.607966    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcefd3df-d067-4b6c-a53e-216c8afb3788-image-registry-private-configuration\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.608187 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.608173    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a37d3e7a-a5f2-465b-b814-31c0dc43842f-cert\") pod \"ingress-canary-7kw4c\" (UID: \"a37d3e7a-a5f2-465b-b814-31c0dc43842f\") " pod="openshift-ingress-canary/ingress-canary-7kw4c"
Jul 28 01:26:27.608298 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.608280    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcefd3df-d067-4b6c-a53e-216c8afb3788-installation-pull-secrets\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.612455 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.612418    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbrtd\" (UniqueName: \"kubernetes.io/projected/a37d3e7a-a5f2-465b-b814-31c0dc43842f-kube-api-access-sbrtd\") pod \"ingress-canary-7kw4c\" (UID: \"a37d3e7a-a5f2-465b-b814-31c0dc43842f\") " pod="openshift-ingress-canary/ingress-canary-7kw4c"
Jul 28 01:26:27.612573 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.612485    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r7h8\" (UniqueName: \"kubernetes.io/projected/aa0f7d7b-c27c-476f-a48c-6a27b3057102-kube-api-access-4r7h8\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:27.612941 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.612921    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78flt\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-kube-api-access-78flt\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.613188 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.613170    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-bound-sa-token\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:27.701129 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.701091    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7kw4c"
Jul 28 01:26:27.987945 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.987915    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-66p5q"]
Jul 28 01:26:27.989797 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.989771    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-66p5q"
Jul 28 01:26:27.992271 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.992238    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9r2w2\""
Jul 28 01:26:27.992399 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.992296    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 01:26:27.992561 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:27.992542    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 01:26:28.006054 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.006021    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-66p5q"]
Jul 28 01:26:28.008186 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.007655    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-88m5j"]
Jul 28 01:26:28.012360 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.012341    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:28.015194 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.014939    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 01:26:28.015194 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.014939    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 01:26:28.015194 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.015017    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 01:26:28.015194 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.015047    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tfwwc\""
Jul 28 01:26:28.015194 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.015141    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 01:26:28.020481 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.020461    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-88m5j"]
Jul 28 01:26:28.107582 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.107549    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:28.107582 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.107585    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:28.107791 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.107613    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdk7x\" (UniqueName: \"kubernetes.io/projected/a594223b-e7f8-403e-92a9-272d7f200e4c-kube-api-access-mdk7x\") pod \"migrator-6865c9967-66p5q\" (UID: \"a594223b-e7f8-403e-92a9-272d7f200e4c\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-66p5q"
Jul 28 01:26:28.107791 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.107645    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-data-volume\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:28.107791 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.107680    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6v95\" (UniqueName: \"kubernetes.io/projected/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-api-access-t6v95\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:28.107791 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.107707    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-crio-socket\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:28.107791 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:28.107710    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 01:26:28.107791 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:28.107731    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8ff4cf8b7-vw2rp: secret "image-registry-tls" not found
Jul 28 01:26:28.107791 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:28.107775    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls podName:fcefd3df-d067-4b6c-a53e-216c8afb3788 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:29.107760924 +0000 UTC m=+36.117279326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls") pod "image-registry-8ff4cf8b7-vw2rp" (UID: "fcefd3df-d067-4b6c-a53e-216c8afb3788") : secret "image-registry-tls" not found
Jul 28 01:26:28.107791 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:28.107708    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:28.108130 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.107830    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:28.108130 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:28.107860    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls podName:aa0f7d7b-c27c-476f-a48c-6a27b3057102 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:29.107842637 +0000 UTC m=+36.117361050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls") pod "dns-default-v6xwj" (UID: "aa0f7d7b-c27c-476f-a48c-6a27b3057102") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:28.108130 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.107931    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:28.208451 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.208414    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdk7x\" (UniqueName: \"kubernetes.io/projected/a594223b-e7f8-403e-92a9-272d7f200e4c-kube-api-access-mdk7x\") pod \"migrator-6865c9967-66p5q\" (UID: \"a594223b-e7f8-403e-92a9-272d7f200e4c\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-66p5q"
Jul 28 01:26:28.208451 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.208454    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-data-volume\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:28.208702 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.208489    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6v95\" (UniqueName: \"kubernetes.io/projected/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-api-access-t6v95\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:28.208702 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.208516    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-crio-socket\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:28.208702 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.208679    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:28.208702 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.208698    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-crio-socket\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:28.209600 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.208733    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:28.209600 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:28.208811    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:28.209600 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:28.208827    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:28.209600 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:28.208924    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm podName:89dc4149-6058-4fe9-90e6-d4f5ef6dd651 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:28.708892272 +0000 UTC m=+35.718410677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88m5j" (UID: "89dc4149-6058-4fe9-90e6-d4f5ef6dd651") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:28.209600 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:28.208971    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls podName:89dc4149-6058-4fe9-90e6-d4f5ef6dd651 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:28.708937077 +0000 UTC m=+35.718455482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88m5j" (UID: "89dc4149-6058-4fe9-90e6-d4f5ef6dd651") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:28.209600 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.209425    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-data-volume\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:28.220212 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.220188    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdk7x\" (UniqueName: \"kubernetes.io/projected/a594223b-e7f8-403e-92a9-272d7f200e4c-kube-api-access-mdk7x\") pod \"migrator-6865c9967-66p5q\" (UID: \"a594223b-e7f8-403e-92a9-272d7f200e4c\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-66p5q"
Jul 28 01:26:28.220212 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.220203    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6v95\" (UniqueName: \"kubernetes.io/projected/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-api-access-t6v95\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:28.300481 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.300417    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-66p5q"
Jul 28 01:26:28.711915 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.711890    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:28.712440 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.711946    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:28.712440 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:28.712075    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:28.712440 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:28.712112    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:28.712440 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:28.712142    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm podName:89dc4149-6058-4fe9-90e6-d4f5ef6dd651 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:29.712117444 +0000 UTC m=+36.721635846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88m5j" (UID: "89dc4149-6058-4fe9-90e6-d4f5ef6dd651") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:28.712440 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:28.712213    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls podName:89dc4149-6058-4fe9-90e6-d4f5ef6dd651 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:29.712163059 +0000 UTC m=+36.721681483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88m5j" (UID: "89dc4149-6058-4fe9-90e6-d4f5ef6dd651") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:28.853976 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.853852    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7kw4c"]
Jul 28 01:26:28.857874 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:26:28.857843    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37d3e7a_a5f2_465b_b814_31c0dc43842f.slice/crio-2397a9fed3f11ecbfdcde3f1406c5b75061bf7bdc24d6c81b54941cf9061b19b WatchSource:0}: Error finding container 2397a9fed3f11ecbfdcde3f1406c5b75061bf7bdc24d6c81b54941cf9061b19b: Status 404 returned error can't find the container with id 2397a9fed3f11ecbfdcde3f1406c5b75061bf7bdc24d6c81b54941cf9061b19b
Jul 28 01:26:28.863119 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:28.863050    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-66p5q"]
Jul 28 01:26:28.868080 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:26:28.868056    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda594223b_e7f8_403e_92a9_272d7f200e4c.slice/crio-13dc2b0fb51c6092efd6bd2991f855adbed8c5efc6f18f085019effd7d3254aa WatchSource:0}: Error finding container 13dc2b0fb51c6092efd6bd2991f855adbed8c5efc6f18f085019effd7d3254aa: Status 404 returned error can't find the container with id 13dc2b0fb51c6092efd6bd2991f855adbed8c5efc6f18f085019effd7d3254aa
Jul 28 01:26:29.114533 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:29.114501    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:29.114664 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:29.114558    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:29.114707 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:29.114663    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:29.114707 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:29.114667    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 01:26:29.114707 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:29.114684    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8ff4cf8b7-vw2rp: secret "image-registry-tls" not found
Jul 28 01:26:29.114805 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:29.114725    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls podName:aa0f7d7b-c27c-476f-a48c-6a27b3057102 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:31.114708792 +0000 UTC m=+38.124227198 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls") pod "dns-default-v6xwj" (UID: "aa0f7d7b-c27c-476f-a48c-6a27b3057102") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:29.114805 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:29.114739    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls podName:fcefd3df-d067-4b6c-a53e-216c8afb3788 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:31.114732871 +0000 UTC m=+38.124251273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls") pod "image-registry-8ff4cf8b7-vw2rp" (UID: "fcefd3df-d067-4b6c-a53e-216c8afb3788") : secret "image-registry-tls" not found
Jul 28 01:26:29.401007 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:29.400929    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k5qjk_80b8c33c-c6e3-4540-ba14-4027f4147c7e/dns-node-resolver/0.log"
Jul 28 01:26:29.721760 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:29.721399    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:29.721760 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:29.721548    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:29.721760 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:29.721602    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:29.721760 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:29.721670    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:29.721760 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:29.721668    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm podName:89dc4149-6058-4fe9-90e6-d4f5ef6dd651 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:31.721652605 +0000 UTC m=+38.731171011 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88m5j" (UID: "89dc4149-6058-4fe9-90e6-d4f5ef6dd651") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:29.721760 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:29.721735    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls podName:89dc4149-6058-4fe9-90e6-d4f5ef6dd651 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:31.721718736 +0000 UTC m=+38.731237143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88m5j" (UID: "89dc4149-6058-4fe9-90e6-d4f5ef6dd651") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:29.735325 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:29.735194    2555 generic.go:356] "Generic (PLEG): container finished" podID="2ba62973-501c-45ad-9710-200326696aa3" containerID="9fa5d727b9d93fdc39efc03a29dba8a45189973bdc301c5f89cc3a3178c3e2bf" exitCode=0
Jul 28 01:26:29.735325 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:29.735269    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8rd2" event={"ID":"2ba62973-501c-45ad-9710-200326696aa3","Type":"ContainerDied","Data":"9fa5d727b9d93fdc39efc03a29dba8a45189973bdc301c5f89cc3a3178c3e2bf"}
Jul 28 01:26:29.736745 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:29.736701    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7kw4c" event={"ID":"a37d3e7a-a5f2-465b-b814-31c0dc43842f","Type":"ContainerStarted","Data":"2397a9fed3f11ecbfdcde3f1406c5b75061bf7bdc24d6c81b54941cf9061b19b"}
Jul 28 01:26:29.737746 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:29.737725    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-66p5q" event={"ID":"a594223b-e7f8-403e-92a9-272d7f200e4c","Type":"ContainerStarted","Data":"13dc2b0fb51c6092efd6bd2991f855adbed8c5efc6f18f085019effd7d3254aa"}
Jul 28 01:26:30.745125 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:30.745088    2555 generic.go:356] "Generic (PLEG): container finished" podID="2ba62973-501c-45ad-9710-200326696aa3" containerID="cae43de08dd9d21dd777021dc648c4d85e84244fac52a1fddf7d37711db8a108" exitCode=0
Jul 28 01:26:30.745631 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:30.745148    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8rd2" event={"ID":"2ba62973-501c-45ad-9710-200326696aa3","Type":"ContainerDied","Data":"cae43de08dd9d21dd777021dc648c4d85e84244fac52a1fddf7d37711db8a108"}
Jul 28 01:26:30.747246 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:30.747172    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-66p5q" event={"ID":"a594223b-e7f8-403e-92a9-272d7f200e4c","Type":"ContainerStarted","Data":"5ba43f8ef8592c31bfd23a3d7e6ab7840b7c7b7af84a77ac00fac71a0981277d"}
Jul 28 01:26:30.747246 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:30.747224    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-66p5q" event={"ID":"a594223b-e7f8-403e-92a9-272d7f200e4c","Type":"ContainerStarted","Data":"2a21261067977b0212b33f5426692ef21e2fa714c684c3e65f71e35a3f745e51"}
Jul 28 01:26:30.799901 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:30.799872    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z9b5j_965314c0-5f5a-4d0e-a133-9fd61ae4c83c/node-ca/0.log"
Jul 28 01:26:31.134111 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:31.133974    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:31.134224 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:31.134114    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:31.134224 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:31.134125    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:31.134224 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:31.134165    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls podName:aa0f7d7b-c27c-476f-a48c-6a27b3057102 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:35.134150196 +0000 UTC m=+42.143668602 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls") pod "dns-default-v6xwj" (UID: "aa0f7d7b-c27c-476f-a48c-6a27b3057102") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:31.134224 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:31.134212    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 01:26:31.134224 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:31.134221    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8ff4cf8b7-vw2rp: secret "image-registry-tls" not found
Jul 28 01:26:31.134409 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:31.134261    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls podName:fcefd3df-d067-4b6c-a53e-216c8afb3788 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:35.134249793 +0000 UTC m=+42.143768195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls") pod "image-registry-8ff4cf8b7-vw2rp" (UID: "fcefd3df-d067-4b6c-a53e-216c8afb3788") : secret "image-registry-tls" not found
Jul 28 01:26:31.739094 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:31.739066    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:31.739234 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:31.739113    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:31.739234 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:31.739219    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:31.739234 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:31.739219    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:31.739321 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:31.739281    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls podName:89dc4149-6058-4fe9-90e6-d4f5ef6dd651 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:35.739264258 +0000 UTC m=+42.748782660 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88m5j" (UID: "89dc4149-6058-4fe9-90e6-d4f5ef6dd651") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:31.739321 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:31.739301    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm podName:89dc4149-6058-4fe9-90e6-d4f5ef6dd651 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:35.739288284 +0000 UTC m=+42.748806686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88m5j" (UID: "89dc4149-6058-4fe9-90e6-d4f5ef6dd651") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:31.752502 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:31.752440    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8rd2" event={"ID":"2ba62973-501c-45ad-9710-200326696aa3","Type":"ContainerStarted","Data":"4181fdf6a16064da19a3e6129991d3628a1aab1610440073ab9e88f202be9804"}
Jul 28 01:26:31.754066 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:31.754040    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7kw4c" event={"ID":"a37d3e7a-a5f2-465b-b814-31c0dc43842f","Type":"ContainerStarted","Data":"866ec14c6e6ee21825fd6d0b8c7c0d51b589521898f19453372e627f54579812"}
Jul 28 01:26:31.779609 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:31.779565    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x8rd2" podStartSLOduration=7.48468151 podStartE2EDuration="37.779551921s" totalImagesPullingTime="30.294870411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:31.77828099 +0000 UTC m=+38.787799432" watchObservedRunningTime="2026-07-28 01:26:31.779551921 +0000 UTC m=+38.789070341"
Jul 28 01:26:31.780085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:31.780059    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-66p5q" podStartSLOduration=3.716100146 podStartE2EDuration="4.780050733s" totalImagesPullingTime="1.063950587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:30.787138671 +0000 UTC m=+37.796657096" watchObservedRunningTime="2026-07-28 01:26:31.780050733 +0000 UTC m=+38.789569156"
Jul 28 01:26:31.796382 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:31.796343    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7kw4c" podStartSLOduration=2.501369698 podStartE2EDuration="4.796328556s" totalImagesPullingTime="2.294958858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:31.796040124 +0000 UTC m=+38.805558550" watchObservedRunningTime="2026-07-28 01:26:31.796328556 +0000 UTC m=+38.805846981"
Jul 28 01:26:35.159501 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:35.159468    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:35.159929 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:35.159507    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:35.159929 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:35.159628    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:35.159929 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:35.159683    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls podName:aa0f7d7b-c27c-476f-a48c-6a27b3057102 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:43.159669671 +0000 UTC m=+50.169188076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls") pod "dns-default-v6xwj" (UID: "aa0f7d7b-c27c-476f-a48c-6a27b3057102") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:35.159929 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:35.159634    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 01:26:35.159929 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:35.159720    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8ff4cf8b7-vw2rp: secret "image-registry-tls" not found
Jul 28 01:26:35.159929 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:35.159758    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls podName:fcefd3df-d067-4b6c-a53e-216c8afb3788 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:43.159746951 +0000 UTC m=+50.169265353 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls") pod "image-registry-8ff4cf8b7-vw2rp" (UID: "fcefd3df-d067-4b6c-a53e-216c8afb3788") : secret "image-registry-tls" not found
Jul 28 01:26:35.764022 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:35.763994    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:35.764184 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:35.764041    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:35.764184 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:35.764130    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:35.764184 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:35.764145    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:35.764184 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:35.764176    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm podName:89dc4149-6058-4fe9-90e6-d4f5ef6dd651 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:43.764163303 +0000 UTC m=+50.773681706 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88m5j" (UID: "89dc4149-6058-4fe9-90e6-d4f5ef6dd651") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:35.764315 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:26:35.764196    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls podName:89dc4149-6058-4fe9-90e6-d4f5ef6dd651 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:43.764183394 +0000 UTC m=+50.773701797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88m5j" (UID: "89dc4149-6058-4fe9-90e6-d4f5ef6dd651") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:43.211871 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.211836    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:43.212283 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.211877    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:43.215823 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.215790    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa0f7d7b-c27c-476f-a48c-6a27b3057102-metrics-tls\") pod \"dns-default-v6xwj\" (UID: \"aa0f7d7b-c27c-476f-a48c-6a27b3057102\") " pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:43.215823 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.215821    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls\") pod \"image-registry-8ff4cf8b7-vw2rp\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") " pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:43.285028 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.285004    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:43.293140 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.293121    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:43.425903 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.425877    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"]
Jul 28 01:26:43.428736 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:26:43.428709    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcefd3df_d067_4b6c_a53e_216c8afb3788.slice/crio-95700861601f3ee12235642260fccea4247bc2c8b65e053414bad3f96e242d2c WatchSource:0}: Error finding container 95700861601f3ee12235642260fccea4247bc2c8b65e053414bad3f96e242d2c: Status 404 returned error can't find the container with id 95700861601f3ee12235642260fccea4247bc2c8b65e053414bad3f96e242d2c
Jul 28 01:26:43.437417 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.437397    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v6xwj"]
Jul 28 01:26:43.440350 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:26:43.440330    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0f7d7b_c27c_476f_a48c_6a27b3057102.slice/crio-cf1591b3e03d00c9be2830db902463f24d2d9aca1959261d5419049d64e9fa86 WatchSource:0}: Error finding container cf1591b3e03d00c9be2830db902463f24d2d9aca1959261d5419049d64e9fa86: Status 404 returned error can't find the container with id cf1591b3e03d00c9be2830db902463f24d2d9aca1959261d5419049d64e9fa86
Jul 28 01:26:43.781613 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.781576    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v6xwj" event={"ID":"aa0f7d7b-c27c-476f-a48c-6a27b3057102","Type":"ContainerStarted","Data":"cf1591b3e03d00c9be2830db902463f24d2d9aca1959261d5419049d64e9fa86"}
Jul 28 01:26:43.783016 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.782993    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp" event={"ID":"fcefd3df-d067-4b6c-a53e-216c8afb3788","Type":"ContainerStarted","Data":"22b9bd46a8e045a5dd7e26bf2c13046737f4ca1b60a21b3a49057656f906b808"}
Jul 28 01:26:43.783132 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.783035    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp" event={"ID":"fcefd3df-d067-4b6c-a53e-216c8afb3788","Type":"ContainerStarted","Data":"95700861601f3ee12235642260fccea4247bc2c8b65e053414bad3f96e242d2c"}
Jul 28 01:26:43.783195 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.783140    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:26:43.801993 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.801933    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp" podStartSLOduration=22.801916206 podStartE2EDuration="22.801916206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:43.800808599 +0000 UTC m=+50.810327047" watchObservedRunningTime="2026-07-28 01:26:43.801916206 +0000 UTC m=+50.811434611"
Jul 28 01:26:43.817879 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.817846    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:43.818496 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.818014    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:43.818496 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.818457    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:43.820768 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.820749    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89dc4149-6058-4fe9-90e6-d4f5ef6dd651-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88m5j\" (UID: \"89dc4149-6058-4fe9-90e6-d4f5ef6dd651\") " pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:43.922048 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:43.922019    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-88m5j"
Jul 28 01:26:44.078105 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:44.078028    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-88m5j"]
Jul 28 01:26:44.081275 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:26:44.081244    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89dc4149_6058_4fe9_90e6_d4f5ef6dd651.slice/crio-edbbe21e8bca66d11c4b23e15a9cc93134d317614757354cb4728fa5eb73c688 WatchSource:0}: Error finding container edbbe21e8bca66d11c4b23e15a9cc93134d317614757354cb4728fa5eb73c688: Status 404 returned error can't find the container with id edbbe21e8bca66d11c4b23e15a9cc93134d317614757354cb4728fa5eb73c688
Jul 28 01:26:44.787612 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:44.787574    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88m5j" event={"ID":"89dc4149-6058-4fe9-90e6-d4f5ef6dd651","Type":"ContainerStarted","Data":"edbbe21e8bca66d11c4b23e15a9cc93134d317614757354cb4728fa5eb73c688"}
Jul 28 01:26:45.792124 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:45.792095    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v6xwj" event={"ID":"aa0f7d7b-c27c-476f-a48c-6a27b3057102","Type":"ContainerStarted","Data":"daff4888702d5913a95c1fb26108f23d5711256d9d4d916df579fd77fdffa9ec"}
Jul 28 01:26:46.797346 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:46.797276    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v6xwj" event={"ID":"aa0f7d7b-c27c-476f-a48c-6a27b3057102","Type":"ContainerStarted","Data":"7d7ba79ab872b10fa2b353c5962522fb90e7280fd564ef2789114195167d21f8"}
Jul 28 01:26:46.797796 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:46.797556    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:46.814653 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:46.814606    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-v6xwj" podStartSLOduration=17.674138713 podStartE2EDuration="19.81459027s" totalImagesPullingTime="2.140451557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:46.813647464 +0000 UTC m=+53.823165908" watchObservedRunningTime="2026-07-28 01:26:46.81459027 +0000 UTC m=+53.824108694"
Jul 28 01:26:49.810280 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:49.810239    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88m5j" event={"ID":"89dc4149-6058-4fe9-90e6-d4f5ef6dd651","Type":"ContainerStarted","Data":"c777cde3811c090378fa1ed9d859bc69ef303a33b268809f41ea0a7de9a30af7"}
Jul 28 01:26:51.435902 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.435871    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"]
Jul 28 01:26:51.440632 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.440601    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln"]
Jul 28 01:26:51.461515 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.461478    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"]
Jul 28 01:26:51.463050 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.463025    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln"
Jul 28 01:26:51.465980 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.465954    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 01:26:51.466218 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.466196    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.467495 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.467322    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 01:26:51.467993 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.467973    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 01:26:51.468582 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.468561    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 01:26:51.469136 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.469117    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 01:26:51.469218 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.469160    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 01:26:51.469379 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.469361    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 01:26:51.470180 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.470159    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f79ddbb4b-245kk"]
Jul 28 01:26:51.471137 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.470996    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 01:26:51.472239 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.472217    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f79ddbb4b-245kk"]
Jul 28 01:26:51.472313 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.472253    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln"]
Jul 28 01:26:51.472313 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.472278    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"]
Jul 28 01:26:51.472416 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.472357    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f79ddbb4b-245kk"
Jul 28 01:26:51.475377 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.475357    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 01:26:51.475464 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.475387    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-kn925\""
Jul 28 01:26:51.478450 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.478428    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf86eae7-c7de-440d-bdd8-8c8dda6868c8-tmp\") pod \"klusterlet-addon-workmgr-84bd54777d-6jvln\" (UID: \"bf86eae7-c7de-440d-bdd8-8c8dda6868c8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln"
Jul 28 01:26:51.478553 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.478468    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bf86eae7-c7de-440d-bdd8-8c8dda6868c8-klusterlet-config\") pod \"klusterlet-addon-workmgr-84bd54777d-6jvln\" (UID: \"bf86eae7-c7de-440d-bdd8-8c8dda6868c8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln"
Jul 28 01:26:51.478553 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.478485    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr8z7\" (UniqueName: \"kubernetes.io/projected/bf86eae7-c7de-440d-bdd8-8c8dda6868c8-kube-api-access-vr8z7\") pod \"klusterlet-addon-workmgr-84bd54777d-6jvln\" (UID: \"bf86eae7-c7de-440d-bdd8-8c8dda6868c8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln"
Jul 28 01:26:51.579684 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.579654    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr8z7\" (UniqueName: \"kubernetes.io/projected/bf86eae7-c7de-440d-bdd8-8c8dda6868c8-kube-api-access-vr8z7\") pod \"klusterlet-addon-workmgr-84bd54777d-6jvln\" (UID: \"bf86eae7-c7de-440d-bdd8-8c8dda6868c8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln"
Jul 28 01:26:51.579857 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.579696    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5765303b-a295-46a5-bd99-b071e4bed13a-ca\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.579857 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.579769    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/95c16a27-c49b-41b1-bbb9-cd0fcf18bacd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f79ddbb4b-245kk\" (UID: \"95c16a27-c49b-41b1-bbb9-cd0fcf18bacd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f79ddbb4b-245kk"
Jul 28 01:26:51.579857 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.579807    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5765303b-a295-46a5-bd99-b071e4bed13a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.579857 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.579839    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf86eae7-c7de-440d-bdd8-8c8dda6868c8-tmp\") pod \"klusterlet-addon-workmgr-84bd54777d-6jvln\" (UID: \"bf86eae7-c7de-440d-bdd8-8c8dda6868c8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln"
Jul 28 01:26:51.580045 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.579863    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bf86eae7-c7de-440d-bdd8-8c8dda6868c8-klusterlet-config\") pod \"klusterlet-addon-workmgr-84bd54777d-6jvln\" (UID: \"bf86eae7-c7de-440d-bdd8-8c8dda6868c8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln"
Jul 28 01:26:51.580045 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.579902    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5765303b-a295-46a5-bd99-b071e4bed13a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.580045 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.579927    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5765303b-a295-46a5-bd99-b071e4bed13a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.580045 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.579948    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrr2j\" (UniqueName: \"kubernetes.io/projected/5765303b-a295-46a5-bd99-b071e4bed13a-kube-api-access-xrr2j\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.580045 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.579973    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmtxb\" (UniqueName: \"kubernetes.io/projected/95c16a27-c49b-41b1-bbb9-cd0fcf18bacd-kube-api-access-vmtxb\") pod \"managed-serviceaccount-addon-agent-f79ddbb4b-245kk\" (UID: \"95c16a27-c49b-41b1-bbb9-cd0fcf18bacd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f79ddbb4b-245kk"
Jul 28 01:26:51.580045 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.579993    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5765303b-a295-46a5-bd99-b071e4bed13a-hub\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.580493 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.580469    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf86eae7-c7de-440d-bdd8-8c8dda6868c8-tmp\") pod \"klusterlet-addon-workmgr-84bd54777d-6jvln\" (UID: \"bf86eae7-c7de-440d-bdd8-8c8dda6868c8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln"
Jul 28 01:26:51.582806 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.582784    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bf86eae7-c7de-440d-bdd8-8c8dda6868c8-klusterlet-config\") pod \"klusterlet-addon-workmgr-84bd54777d-6jvln\" (UID: \"bf86eae7-c7de-440d-bdd8-8c8dda6868c8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln"
Jul 28 01:26:51.628883 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.628847    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr8z7\" (UniqueName: \"kubernetes.io/projected/bf86eae7-c7de-440d-bdd8-8c8dda6868c8-kube-api-access-vr8z7\") pod \"klusterlet-addon-workmgr-84bd54777d-6jvln\" (UID: \"bf86eae7-c7de-440d-bdd8-8c8dda6868c8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln"
Jul 28 01:26:51.680904 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.680877    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5765303b-a295-46a5-bd99-b071e4bed13a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.680904 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.680909    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrr2j\" (UniqueName: \"kubernetes.io/projected/5765303b-a295-46a5-bd99-b071e4bed13a-kube-api-access-xrr2j\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.681119 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.680930    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmtxb\" (UniqueName: \"kubernetes.io/projected/95c16a27-c49b-41b1-bbb9-cd0fcf18bacd-kube-api-access-vmtxb\") pod \"managed-serviceaccount-addon-agent-f79ddbb4b-245kk\" (UID: \"95c16a27-c49b-41b1-bbb9-cd0fcf18bacd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f79ddbb4b-245kk"
Jul 28 01:26:51.681119 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.680946    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5765303b-a295-46a5-bd99-b071e4bed13a-hub\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.681119 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.680969    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5765303b-a295-46a5-bd99-b071e4bed13a-ca\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.681280 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.681119    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/95c16a27-c49b-41b1-bbb9-cd0fcf18bacd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f79ddbb4b-245kk\" (UID: \"95c16a27-c49b-41b1-bbb9-cd0fcf18bacd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f79ddbb4b-245kk"
Jul 28 01:26:51.681280 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.681156    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5765303b-a295-46a5-bd99-b071e4bed13a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.681280 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.681257    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5765303b-a295-46a5-bd99-b071e4bed13a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.682160 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.682123    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5765303b-a295-46a5-bd99-b071e4bed13a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.684080 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.684031    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5765303b-a295-46a5-bd99-b071e4bed13a-ca\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.684185 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.684151    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5765303b-a295-46a5-bd99-b071e4bed13a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.684242 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.684184    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5765303b-a295-46a5-bd99-b071e4bed13a-hub\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.684242 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.684207    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/95c16a27-c49b-41b1-bbb9-cd0fcf18bacd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f79ddbb4b-245kk\" (UID: \"95c16a27-c49b-41b1-bbb9-cd0fcf18bacd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f79ddbb4b-245kk"
Jul 28 01:26:51.684505 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.684489    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5765303b-a295-46a5-bd99-b071e4bed13a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.699266 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.699208    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmtxb\" (UniqueName: \"kubernetes.io/projected/95c16a27-c49b-41b1-bbb9-cd0fcf18bacd-kube-api-access-vmtxb\") pod \"managed-serviceaccount-addon-agent-f79ddbb4b-245kk\" (UID: \"95c16a27-c49b-41b1-bbb9-cd0fcf18bacd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f79ddbb4b-245kk"
Jul 28 01:26:51.700968 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.700943    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrr2j\" (UniqueName: \"kubernetes.io/projected/5765303b-a295-46a5-bd99-b071e4bed13a-kube-api-access-xrr2j\") pod \"cluster-proxy-proxy-agent-74cf56b797-f9t9m\" (UID: \"5765303b-a295-46a5-bd99-b071e4bed13a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.776140 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.776113    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln"
Jul 28 01:26:51.811236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.811207    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"
Jul 28 01:26:51.816618 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:51.816597    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f79ddbb4b-245kk"
Jul 28 01:26:52.824835 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:52.824796    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln"]
Jul 28 01:26:52.827328 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:26:52.827287    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf86eae7_c7de_440d_bdd8_8c8dda6868c8.slice/crio-8d13d10bc4d8e60664f42eb4a5b8829253a98da9a2f1b23b4233a2af686cf2a8 WatchSource:0}: Error finding container 8d13d10bc4d8e60664f42eb4a5b8829253a98da9a2f1b23b4233a2af686cf2a8: Status 404 returned error can't find the container with id 8d13d10bc4d8e60664f42eb4a5b8829253a98da9a2f1b23b4233a2af686cf2a8
Jul 28 01:26:53.041353 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:53.041326    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f79ddbb4b-245kk"]
Jul 28 01:26:53.044310 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:26:53.044261    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95c16a27_c49b_41b1_bbb9_cd0fcf18bacd.slice/crio-c19333c6bc3277f06a2dc06137b001d894f04e204e8fbd5b146a52c458a4f0fa WatchSource:0}: Error finding container c19333c6bc3277f06a2dc06137b001d894f04e204e8fbd5b146a52c458a4f0fa: Status 404 returned error can't find the container with id c19333c6bc3277f06a2dc06137b001d894f04e204e8fbd5b146a52c458a4f0fa
Jul 28 01:26:53.044510 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:53.044492    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m"]
Jul 28 01:26:53.742874 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:53.742845    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54bw2"
Jul 28 01:26:53.827473 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:53.827438    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln" event={"ID":"bf86eae7-c7de-440d-bdd8-8c8dda6868c8","Type":"ContainerStarted","Data":"8d13d10bc4d8e60664f42eb4a5b8829253a98da9a2f1b23b4233a2af686cf2a8"}
Jul 28 01:26:53.829645 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:53.829615    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f79ddbb4b-245kk" event={"ID":"95c16a27-c49b-41b1-bbb9-cd0fcf18bacd","Type":"ContainerStarted","Data":"c19333c6bc3277f06a2dc06137b001d894f04e204e8fbd5b146a52c458a4f0fa"}
Jul 28 01:26:53.836969 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:53.836939    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88m5j" event={"ID":"89dc4149-6058-4fe9-90e6-d4f5ef6dd651","Type":"ContainerStarted","Data":"f983c857b1f05847ee6a63c5a7d4207784c0a6e536021a5e22a818e64afd6402"}
Jul 28 01:26:53.842458 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:53.842431    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m" event={"ID":"5765303b-a295-46a5-bd99-b071e4bed13a","Type":"ContainerStarted","Data":"e979a0b868ea5c162343bcfaf5a0e9c8f9dd332ce69fe5dd27d1f284d03d534d"}
Jul 28 01:26:56.804002 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:56.803968    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v6xwj"
Jul 28 01:26:58.136029 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:58.135995    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret\") pod \"global-pull-secret-syncer-f4xdf\" (UID: \"be56d98e-e52b-4ea8-9187-511176665495\") " pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:58.138747 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:58.138725    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 01:26:58.148905 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:58.148875    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be56d98e-e52b-4ea8-9187-511176665495-original-pull-secret\") pod \"global-pull-secret-syncer-f4xdf\" (UID: \"be56d98e-e52b-4ea8-9187-511176665495\") " pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:58.237163 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:58.237130    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs\") pod \"network-metrics-daemon-k24x4\" (UID: \"3442c3b8-9ee2-4ad0-bbcf-b0918932580e\") " pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:58.239985 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:58.239962    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 01:26:58.250563 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:58.250511    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3442c3b8-9ee2-4ad0-bbcf-b0918932580e-metrics-certs\") pod \"network-metrics-daemon-k24x4\" (UID: \"3442c3b8-9ee2-4ad0-bbcf-b0918932580e\") " pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:58.338224 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:58.338186    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhg9\" (UniqueName: \"kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9\") pod \"network-check-target-q8gnk\" (UID: \"4ba73e34-6176-472c-bbde-ffd0c9b335eb\") " pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:58.340906 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:58.340887    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 01:26:58.351539 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:58.351497    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 01:26:58.362407 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:58.362385    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmhg9\" (UniqueName: \"kubernetes.io/projected/4ba73e34-6176-472c-bbde-ffd0c9b335eb-kube-api-access-hmhg9\") pod \"network-check-target-q8gnk\" (UID: \"4ba73e34-6176-472c-bbde-ffd0c9b335eb\") " pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:26:58.365108 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:58.365087    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xdf"
Jul 28 01:26:58.373380 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:58.373362    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ptgdn\""
Jul 28 01:26:58.381493 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:58.381473    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k24x4"
Jul 28 01:26:58.661146 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:58.661115    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6xpqm\""
Jul 28 01:26:58.669258 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:26:58.669234    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:27:01.387754 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.387731    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f4xdf"]
Jul 28 01:27:01.392321 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:27:01.392274    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe56d98e_e52b_4ea8_9187_511176665495.slice/crio-ab2a39dc1e30c70065855f0ba6cde9105c09e525e3d253c6b711d49bb9e7d922 WatchSource:0}: Error finding container ab2a39dc1e30c70065855f0ba6cde9105c09e525e3d253c6b711d49bb9e7d922: Status 404 returned error can't find the container with id ab2a39dc1e30c70065855f0ba6cde9105c09e525e3d253c6b711d49bb9e7d922
Jul 28 01:27:01.442138 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.442109    2555 patch_prober.go:28] interesting pod/image-registry-8ff4cf8b7-vw2rp container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 01:27:01.442278 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.442153    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp" podUID="fcefd3df-d067-4b6c-a53e-216c8afb3788" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:27:01.608709 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.608679    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k24x4"]
Jul 28 01:27:01.611333 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.611288    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q8gnk"]
Jul 28 01:27:01.613015 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:27:01.612990    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3442c3b8_9ee2_4ad0_bbcf_b0918932580e.slice/crio-6021a1ee337b91a79ebe2bbaeb0143f0c7f067ebfca30f71b6719965f89fc316 WatchSource:0}: Error finding container 6021a1ee337b91a79ebe2bbaeb0143f0c7f067ebfca30f71b6719965f89fc316: Status 404 returned error can't find the container with id 6021a1ee337b91a79ebe2bbaeb0143f0c7f067ebfca30f71b6719965f89fc316
Jul 28 01:27:01.614662 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:27:01.614640    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba73e34_6176_472c_bbde_ffd0c9b335eb.slice/crio-4878e0a4164c4b4bcb233cd9ed9d357b1b409612993feaf077084d8fc1235a11 WatchSource:0}: Error finding container 4878e0a4164c4b4bcb233cd9ed9d357b1b409612993feaf077084d8fc1235a11: Status 404 returned error can't find the container with id 4878e0a4164c4b4bcb233cd9ed9d357b1b409612993feaf077084d8fc1235a11
Jul 28 01:27:01.880000 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.879968    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m" event={"ID":"5765303b-a295-46a5-bd99-b071e4bed13a","Type":"ContainerStarted","Data":"872dbb54f3ab3da44a8c863fea2fa0e03d7c666335abee9f20b818063315859a"}
Jul 28 01:27:01.880000 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.880021    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m" event={"ID":"5765303b-a295-46a5-bd99-b071e4bed13a","Type":"ContainerStarted","Data":"52be2748427f704617bc0854ded8d876134e81d11e0bbd2343076af1c084a24b"}
Jul 28 01:27:01.880000 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.880043    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m" event={"ID":"5765303b-a295-46a5-bd99-b071e4bed13a","Type":"ContainerStarted","Data":"695a1f97a846668298c20348a9dbaf418cd9025ae9286c37c30d7ada10767345"}
Jul 28 01:27:01.881298 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.881269    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q8gnk" event={"ID":"4ba73e34-6176-472c-bbde-ffd0c9b335eb","Type":"ContainerStarted","Data":"4878e0a4164c4b4bcb233cd9ed9d357b1b409612993feaf077084d8fc1235a11"}
Jul 28 01:27:01.882656 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.882631    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k24x4" event={"ID":"3442c3b8-9ee2-4ad0-bbcf-b0918932580e","Type":"ContainerStarted","Data":"6021a1ee337b91a79ebe2bbaeb0143f0c7f067ebfca30f71b6719965f89fc316"}
Jul 28 01:27:01.884262 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.884235    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln" event={"ID":"bf86eae7-c7de-440d-bdd8-8c8dda6868c8","Type":"ContainerStarted","Data":"f1da5ed4e02d0fb0961d105173b9b21f6a76a5b230ba80c309cd444b256308b2"}
Jul 28 01:27:01.884439 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.884421    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln"
Jul 28 01:27:01.886458 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.886415    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f79ddbb4b-245kk" event={"ID":"95c16a27-c49b-41b1-bbb9-cd0fcf18bacd","Type":"ContainerStarted","Data":"d2086a8e32032d24473c145c1815cf578365413c4783c3a42a415e8b9341fbce"}
Jul 28 01:27:01.886599 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.886514    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln"
Jul 28 01:27:01.888914 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.888895    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88m5j" event={"ID":"89dc4149-6058-4fe9-90e6-d4f5ef6dd651","Type":"ContainerStarted","Data":"2ef735512b3f255c10bb3c1383b6eca2e0b10d33d8fb9b170c8215991f3ad002"}
Jul 28 01:27:01.890186 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.890139    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f4xdf" event={"ID":"be56d98e-e52b-4ea8-9187-511176665495","Type":"ContainerStarted","Data":"ab2a39dc1e30c70065855f0ba6cde9105c09e525e3d253c6b711d49bb9e7d922"}
Jul 28 01:27:01.901450 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.901409    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74cf56b797-f9t9m" podStartSLOduration=2.720465506 podStartE2EDuration="10.901393497s" totalImagesPullingTime="8.180927991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:01.900177311 +0000 UTC m=+68.909695737" watchObservedRunningTime="2026-07-28 01:27:01.901393497 +0000 UTC m=+68.910911923"
Jul 28 01:27:01.916140 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.915647    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd54777d-6jvln" podStartSLOduration=2.494222953 podStartE2EDuration="10.915634302s" totalImagesPullingTime="8.421411349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:01.914742383 +0000 UTC m=+68.924260809" watchObservedRunningTime="2026-07-28 01:27:01.915634302 +0000 UTC m=+68.925152704"
Jul 28 01:27:01.933426 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.933386    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-88m5j" podStartSLOduration=17.974693034 podStartE2EDuration="34.933375125s" totalImagesPullingTime="16.958682091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:01.932657512 +0000 UTC m=+68.942175953" watchObservedRunningTime="2026-07-28 01:27:01.933375125 +0000 UTC m=+68.942893548"
Jul 28 01:27:01.979853 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:01.979806    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f79ddbb4b-245kk" podStartSLOduration=2.799121247 podStartE2EDuration="10.97979033s" totalImagesPullingTime="8.180669083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:01.978412332 +0000 UTC m=+68.987930756" watchObservedRunningTime="2026-07-28 01:27:01.97979033 +0000 UTC m=+68.989308754"
Jul 28 01:27:02.896503 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:02.896461    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k24x4" event={"ID":"3442c3b8-9ee2-4ad0-bbcf-b0918932580e","Type":"ContainerStarted","Data":"a5b978e131dc10f31db2d4c4b0348a3ee2860dc3bccd75f77477a22cbad153ad"}
Jul 28 01:27:03.902126 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:03.902084    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k24x4" event={"ID":"3442c3b8-9ee2-4ad0-bbcf-b0918932580e","Type":"ContainerStarted","Data":"4d352648d697274b9ed79ab5cf9ccc050180706b40c37f9c56607c035b3fae0f"}
Jul 28 01:27:03.917762 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:03.917716    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-k24x4" podStartSLOduration=68.806866505 podStartE2EDuration="1m9.917701505s" totalImagesPullingTime="1.110835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:03.91743351 +0000 UTC m=+70.926951936" watchObservedRunningTime="2026-07-28 01:27:03.917701505 +0000 UTC m=+70.927219928"
Jul 28 01:27:06.915242 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:06.915206    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q8gnk" event={"ID":"4ba73e34-6176-472c-bbde-ffd0c9b335eb","Type":"ContainerStarted","Data":"e25e4b3cbbef1954ec242ba0d34ece7d10c872c0972b9aa0d07f6cf54f82a770"}
Jul 28 01:27:06.915799 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:06.915270    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:27:06.916846 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:06.916819    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f4xdf" event={"ID":"be56d98e-e52b-4ea8-9187-511176665495","Type":"ContainerStarted","Data":"b1d837cdf974974a7c7c684a57a62d377d40d518fc1d78c8792e6b1b0ae46ff3"}
Jul 28 01:27:06.933645 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:06.933597    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q8gnk" podStartSLOduration=68.484392067 podStartE2EDuration="1m12.933582522s" totalImagesPullingTime="4.449190455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:06.932078324 +0000 UTC m=+73.941596758" watchObservedRunningTime="2026-07-28 01:27:06.933582522 +0000 UTC m=+73.943100947"
Jul 28 01:27:06.947153 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:06.947117    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f4xdf" podStartSLOduration=68.272103964 podStartE2EDuration="1m12.947102909s" totalImagesPullingTime="4.674998945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:06.946496439 +0000 UTC m=+73.956014873" watchObservedRunningTime="2026-07-28 01:27:06.947102909 +0000 UTC m=+73.956621332"
Jul 28 01:27:11.441005 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:11.440975    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:27:16.456893 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.456736    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp" podUID="fcefd3df-d067-4b6c-a53e-216c8afb3788" containerName="registry" containerID="cri-o://22b9bd46a8e045a5dd7e26bf2c13046737f4ca1b60a21b3a49057656f906b808" gracePeriod=30
Jul 28 01:27:16.701171 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.701151    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:27:16.768490 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.768464    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcefd3df-d067-4b6c-a53e-216c8afb3788-installation-pull-secrets\") pod \"fcefd3df-d067-4b6c-a53e-216c8afb3788\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") "
Jul 28 01:27:16.768647 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.768499    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcefd3df-d067-4b6c-a53e-216c8afb3788-image-registry-private-configuration\") pod \"fcefd3df-d067-4b6c-a53e-216c8afb3788\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") "
Jul 28 01:27:16.768647 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.768560    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-bound-sa-token\") pod \"fcefd3df-d067-4b6c-a53e-216c8afb3788\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") "
Jul 28 01:27:16.768647 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.768587    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-certificates\") pod \"fcefd3df-d067-4b6c-a53e-216c8afb3788\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") "
Jul 28 01:27:16.768647 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.768610    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcefd3df-d067-4b6c-a53e-216c8afb3788-trusted-ca\") pod \"fcefd3df-d067-4b6c-a53e-216c8afb3788\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") "
Jul 28 01:27:16.768647 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.768644    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls\") pod \"fcefd3df-d067-4b6c-a53e-216c8afb3788\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") "
Jul 28 01:27:16.768894 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.768674    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcefd3df-d067-4b6c-a53e-216c8afb3788-ca-trust-extracted\") pod \"fcefd3df-d067-4b6c-a53e-216c8afb3788\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") "
Jul 28 01:27:16.768894 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.768706    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78flt\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-kube-api-access-78flt\") pod \"fcefd3df-d067-4b6c-a53e-216c8afb3788\" (UID: \"fcefd3df-d067-4b6c-a53e-216c8afb3788\") "
Jul 28 01:27:16.769067 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.769039    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fcefd3df-d067-4b6c-a53e-216c8afb3788" (UID: "fcefd3df-d067-4b6c-a53e-216c8afb3788"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:27:16.769374 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.769067    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcefd3df-d067-4b6c-a53e-216c8afb3788-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fcefd3df-d067-4b6c-a53e-216c8afb3788" (UID: "fcefd3df-d067-4b6c-a53e-216c8afb3788"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:27:16.772497 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.772468    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcefd3df-d067-4b6c-a53e-216c8afb3788-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fcefd3df-d067-4b6c-a53e-216c8afb3788" (UID: "fcefd3df-d067-4b6c-a53e-216c8afb3788"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:27:16.772626 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.772480    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-kube-api-access-78flt" (OuterVolumeSpecName: "kube-api-access-78flt") pod "fcefd3df-d067-4b6c-a53e-216c8afb3788" (UID: "fcefd3df-d067-4b6c-a53e-216c8afb3788"). InnerVolumeSpecName "kube-api-access-78flt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:27:16.772626 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.772593    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fcefd3df-d067-4b6c-a53e-216c8afb3788" (UID: "fcefd3df-d067-4b6c-a53e-216c8afb3788"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:27:16.772735 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.772628    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcefd3df-d067-4b6c-a53e-216c8afb3788-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fcefd3df-d067-4b6c-a53e-216c8afb3788" (UID: "fcefd3df-d067-4b6c-a53e-216c8afb3788"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:27:16.772735 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.772662    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fcefd3df-d067-4b6c-a53e-216c8afb3788" (UID: "fcefd3df-d067-4b6c-a53e-216c8afb3788"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:27:16.778798 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.778767    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcefd3df-d067-4b6c-a53e-216c8afb3788-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fcefd3df-d067-4b6c-a53e-216c8afb3788" (UID: "fcefd3df-d067-4b6c-a53e-216c8afb3788"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:27:16.869269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.869228    2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-bound-sa-token\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:27:16.869269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.869265    2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-certificates\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:27:16.869269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.869278    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcefd3df-d067-4b6c-a53e-216c8afb3788-trusted-ca\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:27:16.869269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.869286    2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-registry-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:27:16.869482 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.869293    2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcefd3df-d067-4b6c-a53e-216c8afb3788-ca-trust-extracted\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:27:16.869482 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.869301    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78flt\" (UniqueName: \"kubernetes.io/projected/fcefd3df-d067-4b6c-a53e-216c8afb3788-kube-api-access-78flt\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:27:16.869482 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.869309    2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcefd3df-d067-4b6c-a53e-216c8afb3788-installation-pull-secrets\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:27:16.869482 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.869318    2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcefd3df-d067-4b6c-a53e-216c8afb3788-image-registry-private-configuration\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:27:16.948343 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.948312    2555 generic.go:356] "Generic (PLEG): container finished" podID="fcefd3df-d067-4b6c-a53e-216c8afb3788" containerID="22b9bd46a8e045a5dd7e26bf2c13046737f4ca1b60a21b3a49057656f906b808" exitCode=0
Jul 28 01:27:16.948503 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.948362    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"
Jul 28 01:27:16.948503 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.948396    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp" event={"ID":"fcefd3df-d067-4b6c-a53e-216c8afb3788","Type":"ContainerDied","Data":"22b9bd46a8e045a5dd7e26bf2c13046737f4ca1b60a21b3a49057656f906b808"}
Jul 28 01:27:16.948503 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.948447    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8ff4cf8b7-vw2rp" event={"ID":"fcefd3df-d067-4b6c-a53e-216c8afb3788","Type":"ContainerDied","Data":"95700861601f3ee12235642260fccea4247bc2c8b65e053414bad3f96e242d2c"}
Jul 28 01:27:16.948503 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.948469    2555 scope.go:117] "RemoveContainer" containerID="22b9bd46a8e045a5dd7e26bf2c13046737f4ca1b60a21b3a49057656f906b808"
Jul 28 01:27:16.957238 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.957224    2555 scope.go:117] "RemoveContainer" containerID="22b9bd46a8e045a5dd7e26bf2c13046737f4ca1b60a21b3a49057656f906b808"
Jul 28 01:27:16.957477 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:27:16.957457    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b9bd46a8e045a5dd7e26bf2c13046737f4ca1b60a21b3a49057656f906b808\": container with ID starting with 22b9bd46a8e045a5dd7e26bf2c13046737f4ca1b60a21b3a49057656f906b808 not found: ID does not exist" containerID="22b9bd46a8e045a5dd7e26bf2c13046737f4ca1b60a21b3a49057656f906b808"
Jul 28 01:27:16.957552 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.957484    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b9bd46a8e045a5dd7e26bf2c13046737f4ca1b60a21b3a49057656f906b808"} err="failed to get container status \"22b9bd46a8e045a5dd7e26bf2c13046737f4ca1b60a21b3a49057656f906b808\": rpc error: code = NotFound desc = could not find container \"22b9bd46a8e045a5dd7e26bf2c13046737f4ca1b60a21b3a49057656f906b808\": container with ID starting with 22b9bd46a8e045a5dd7e26bf2c13046737f4ca1b60a21b3a49057656f906b808 not found: ID does not exist"
Jul 28 01:27:16.968198 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.968178    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"]
Jul 28 01:27:16.971543 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:16.971512    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8ff4cf8b7-vw2rp"]
Jul 28 01:27:17.451759 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:17.451730    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcefd3df-d067-4b6c-a53e-216c8afb3788" path="/var/lib/kubelet/pods/fcefd3df-d067-4b6c-a53e-216c8afb3788/volumes"
Jul 28 01:27:24.520145 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.520114    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m8x8k"]
Jul 28 01:27:24.520613 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.520307    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcefd3df-d067-4b6c-a53e-216c8afb3788" containerName="registry"
Jul 28 01:27:24.520613 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.520317    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcefd3df-d067-4b6c-a53e-216c8afb3788" containerName="registry"
Jul 28 01:27:24.520613 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.520369    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcefd3df-d067-4b6c-a53e-216c8afb3788" containerName="registry"
Jul 28 01:27:24.553507 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.553484    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.556399 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.556372    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 01:27:24.557390 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.557361    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 01:27:24.558400 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.557978    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 01:27:24.558400 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.558049    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 01:27:24.558400 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.558289    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-469rd\""
Jul 28 01:27:24.558400 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.558601    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 01:27:24.560595 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.559608    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 01:27:24.622249 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.622225    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ce30f949-8274-4b0f-b59c-91ab0f311dde-node-exporter-wtmp\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.622363 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.622257    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ce30f949-8274-4b0f-b59c-91ab0f311dde-node-exporter-accelerators-collector-config\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.622363 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.622278    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ce30f949-8274-4b0f-b59c-91ab0f311dde-root\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.622363 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.622294    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce30f949-8274-4b0f-b59c-91ab0f311dde-sys\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.622485 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.622373    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ce30f949-8274-4b0f-b59c-91ab0f311dde-node-exporter-textfile\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.622485 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.622415    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce30f949-8274-4b0f-b59c-91ab0f311dde-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.622485 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.622437    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce30f949-8274-4b0f-b59c-91ab0f311dde-metrics-client-ca\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.622485 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.622453    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ce30f949-8274-4b0f-b59c-91ab0f311dde-node-exporter-tls\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.622485 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.622475    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dz77\" (UniqueName: \"kubernetes.io/projected/ce30f949-8274-4b0f-b59c-91ab0f311dde-kube-api-access-5dz77\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.723255 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.723226    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ce30f949-8274-4b0f-b59c-91ab0f311dde-root\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.723375 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.723262    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce30f949-8274-4b0f-b59c-91ab0f311dde-sys\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.723375 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.723295    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ce30f949-8274-4b0f-b59c-91ab0f311dde-node-exporter-textfile\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.723375 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.723323    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce30f949-8274-4b0f-b59c-91ab0f311dde-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.723375 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.723335    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ce30f949-8274-4b0f-b59c-91ab0f311dde-root\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.723375 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.723350    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce30f949-8274-4b0f-b59c-91ab0f311dde-metrics-client-ca\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.723375 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.723337    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce30f949-8274-4b0f-b59c-91ab0f311dde-sys\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.723375 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.723374    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ce30f949-8274-4b0f-b59c-91ab0f311dde-node-exporter-tls\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.723755 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.723399    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dz77\" (UniqueName: \"kubernetes.io/projected/ce30f949-8274-4b0f-b59c-91ab0f311dde-kube-api-access-5dz77\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.723755 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.723438    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ce30f949-8274-4b0f-b59c-91ab0f311dde-node-exporter-wtmp\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.723755 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.723476    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ce30f949-8274-4b0f-b59c-91ab0f311dde-node-exporter-accelerators-collector-config\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.723755 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.723644    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ce30f949-8274-4b0f-b59c-91ab0f311dde-node-exporter-textfile\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.723755 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.723667    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ce30f949-8274-4b0f-b59c-91ab0f311dde-node-exporter-wtmp\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.723998 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.723960    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce30f949-8274-4b0f-b59c-91ab0f311dde-metrics-client-ca\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.724103 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.724084    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ce30f949-8274-4b0f-b59c-91ab0f311dde-node-exporter-accelerators-collector-config\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.725724 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.725705    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ce30f949-8274-4b0f-b59c-91ab0f311dde-node-exporter-tls\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.725805 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.725735    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce30f949-8274-4b0f-b59c-91ab0f311dde-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.731880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.731860    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dz77\" (UniqueName: \"kubernetes.io/projected/ce30f949-8274-4b0f-b59c-91ab0f311dde-kube-api-access-5dz77\") pod \"node-exporter-m8x8k\" (UID: \"ce30f949-8274-4b0f-b59c-91ab0f311dde\") " pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.864301 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.864232    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m8x8k"
Jul 28 01:27:24.875470 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:27:24.875444    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce30f949_8274_4b0f_b59c_91ab0f311dde.slice/crio-95843abdaba1070e2a10c33f569af5d86290930b6901453f0aba652525c4e6af WatchSource:0}: Error finding container 95843abdaba1070e2a10c33f569af5d86290930b6901453f0aba652525c4e6af: Status 404 returned error can't find the container with id 95843abdaba1070e2a10c33f569af5d86290930b6901453f0aba652525c4e6af
Jul 28 01:27:24.975620 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:24.975592    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m8x8k" event={"ID":"ce30f949-8274-4b0f-b59c-91ab0f311dde","Type":"ContainerStarted","Data":"95843abdaba1070e2a10c33f569af5d86290930b6901453f0aba652525c4e6af"}
Jul 28 01:27:25.979866 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:25.979835    2555 generic.go:356] "Generic (PLEG): container finished" podID="ce30f949-8274-4b0f-b59c-91ab0f311dde" containerID="31cf6d70ea0007c544742a9cff7725d6384157f78c3ac429641da8e8bc973638" exitCode=0
Jul 28 01:27:25.980259 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:25.979899    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m8x8k" event={"ID":"ce30f949-8274-4b0f-b59c-91ab0f311dde","Type":"ContainerDied","Data":"31cf6d70ea0007c544742a9cff7725d6384157f78c3ac429641da8e8bc973638"}
Jul 28 01:27:26.984468 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:26.984431    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m8x8k" event={"ID":"ce30f949-8274-4b0f-b59c-91ab0f311dde","Type":"ContainerStarted","Data":"fd9124da9920eb9333153841a64abfdf0524750147f59a8f664989d9233de7d9"}
Jul 28 01:27:26.984934 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:26.984481    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m8x8k" event={"ID":"ce30f949-8274-4b0f-b59c-91ab0f311dde","Type":"ContainerStarted","Data":"f12e915262608c7796c332879ca918e1c1602ab18c541f7fad7289e74950a7d3"}
Jul 28 01:27:27.006464 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:27.006412    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m8x8k" podStartSLOduration=2.080031531 podStartE2EDuration="3.006396509s" totalImagesPullingTime="857.473315ms" totalInitContainerRuntime="68.891663ms" isStatefulPod=true podCreationTimestamp="2026-07-28 01:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:27.00476506 +0000 UTC m=+94.014283499" watchObservedRunningTime="2026-07-28 01:27:27.006396509 +0000 UTC m=+94.015914955"
Jul 28 01:27:37.922947 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:37.922918    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q8gnk"
Jul 28 01:27:37.932973 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:27:37.932942    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7kw4c_a37d3e7a-a5f2-465b-b814-31c0dc43842f/serve-healthcheck-canary/0.log"
Jul 28 01:30:53.333805 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:30:53.333778    2555 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 01:33:34.653315 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.653243    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-sc9gn"]
Jul 28 01:33:34.655238 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.655222    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-sc9gn"
Jul 28 01:33:34.658042 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.658011    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 01:33:34.658166 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.658016    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 01:33:34.658166 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.658052    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 01:33:34.659060 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.659044    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-wn7j6\""
Jul 28 01:33:34.665365 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.665346    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-sc9gn"]
Jul 28 01:33:34.668803 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.668786    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-tvgjt"]
Jul 28 01:33:34.670924 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.670908    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-tvgjt"
Jul 28 01:33:34.671369 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.671348    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3-tls-certs\") pod \"model-serving-api-7b954ffc95-sc9gn\" (UID: \"9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3\") " pod="kserve/model-serving-api-7b954ffc95-sc9gn"
Jul 28 01:33:34.671485 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.671468    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htpjk\" (UniqueName: \"kubernetes.io/projected/9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3-kube-api-access-htpjk\") pod \"model-serving-api-7b954ffc95-sc9gn\" (UID: \"9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3\") " pod="kserve/model-serving-api-7b954ffc95-sc9gn"
Jul 28 01:33:34.673786 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.673766    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-2klb4\""
Jul 28 01:33:34.673868 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.673818    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 01:33:34.681919 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.681895    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-tvgjt"]
Jul 28 01:33:34.772245 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.772218    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htpjk\" (UniqueName: \"kubernetes.io/projected/9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3-kube-api-access-htpjk\") pod \"model-serving-api-7b954ffc95-sc9gn\" (UID: \"9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3\") " pod="kserve/model-serving-api-7b954ffc95-sc9gn"
Jul 28 01:33:34.772349 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.772256    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3-tls-certs\") pod \"model-serving-api-7b954ffc95-sc9gn\" (UID: \"9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3\") " pod="kserve/model-serving-api-7b954ffc95-sc9gn"
Jul 28 01:33:34.772349 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.772276    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwvjm\" (UniqueName: \"kubernetes.io/projected/0d3eac21-7291-4d7e-8209-4d55b8a88f22-kube-api-access-vwvjm\") pod \"odh-model-controller-75d9cdbddf-tvgjt\" (UID: \"0d3eac21-7291-4d7e-8209-4d55b8a88f22\") " pod="kserve/odh-model-controller-75d9cdbddf-tvgjt"
Jul 28 01:33:34.772349 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.772294    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d3eac21-7291-4d7e-8209-4d55b8a88f22-cert\") pod \"odh-model-controller-75d9cdbddf-tvgjt\" (UID: \"0d3eac21-7291-4d7e-8209-4d55b8a88f22\") " pod="kserve/odh-model-controller-75d9cdbddf-tvgjt"
Jul 28 01:33:34.772450 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:33:34.772384    2555 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 01:33:34.772481 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:33:34.772467    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3-tls-certs podName:9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3 nodeName:}" failed. No retries permitted until 2026-07-28 01:33:35.272450913 +0000 UTC m=+462.281969315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3-tls-certs") pod "model-serving-api-7b954ffc95-sc9gn" (UID: "9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3") : secret "model-serving-api-tls" not found
Jul 28 01:33:34.781133 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.781105    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htpjk\" (UniqueName: \"kubernetes.io/projected/9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3-kube-api-access-htpjk\") pod \"model-serving-api-7b954ffc95-sc9gn\" (UID: \"9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3\") " pod="kserve/model-serving-api-7b954ffc95-sc9gn"
Jul 28 01:33:34.873080 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.873055    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwvjm\" (UniqueName: \"kubernetes.io/projected/0d3eac21-7291-4d7e-8209-4d55b8a88f22-kube-api-access-vwvjm\") pod \"odh-model-controller-75d9cdbddf-tvgjt\" (UID: \"0d3eac21-7291-4d7e-8209-4d55b8a88f22\") " pod="kserve/odh-model-controller-75d9cdbddf-tvgjt"
Jul 28 01:33:34.873175 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.873084    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d3eac21-7291-4d7e-8209-4d55b8a88f22-cert\") pod \"odh-model-controller-75d9cdbddf-tvgjt\" (UID: \"0d3eac21-7291-4d7e-8209-4d55b8a88f22\") " pod="kserve/odh-model-controller-75d9cdbddf-tvgjt"
Jul 28 01:33:34.875332 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.875312    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d3eac21-7291-4d7e-8209-4d55b8a88f22-cert\") pod \"odh-model-controller-75d9cdbddf-tvgjt\" (UID: \"0d3eac21-7291-4d7e-8209-4d55b8a88f22\") " pod="kserve/odh-model-controller-75d9cdbddf-tvgjt"
Jul 28 01:33:34.883799 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.883774    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwvjm\" (UniqueName: \"kubernetes.io/projected/0d3eac21-7291-4d7e-8209-4d55b8a88f22-kube-api-access-vwvjm\") pod \"odh-model-controller-75d9cdbddf-tvgjt\" (UID: \"0d3eac21-7291-4d7e-8209-4d55b8a88f22\") " pod="kserve/odh-model-controller-75d9cdbddf-tvgjt"
Jul 28 01:33:34.980340 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:34.980282    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-tvgjt"
Jul 28 01:33:35.098654 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:35.098632    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-tvgjt"]
Jul 28 01:33:35.100846 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:33:35.100821    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3eac21_7291_4d7e_8209_4d55b8a88f22.slice/crio-f5e22080f2829a62bc829147a71b84f6b7bdee5e0f9efd79e460b6eee0c2aad2 WatchSource:0}: Error finding container f5e22080f2829a62bc829147a71b84f6b7bdee5e0f9efd79e460b6eee0c2aad2: Status 404 returned error can't find the container with id f5e22080f2829a62bc829147a71b84f6b7bdee5e0f9efd79e460b6eee0c2aad2
Jul 28 01:33:35.102119 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:35.102100    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:33:35.276923 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:35.276894    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3-tls-certs\") pod \"model-serving-api-7b954ffc95-sc9gn\" (UID: \"9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3\") " pod="kserve/model-serving-api-7b954ffc95-sc9gn"
Jul 28 01:33:35.279253 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:35.279236    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3-tls-certs\") pod \"model-serving-api-7b954ffc95-sc9gn\" (UID: \"9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3\") " pod="kserve/model-serving-api-7b954ffc95-sc9gn"
Jul 28 01:33:35.564848 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:35.564786    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-sc9gn"
Jul 28 01:33:35.691845 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:35.691812    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-sc9gn"]
Jul 28 01:33:35.694885 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:33:35.694855    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9855f8c7_1a90_4a2b_90f9_e9c1fad8c1b3.slice/crio-1d12c3480400cddedaaa11868e36eff35ebecbc483e8b37c39037ecd588cbc61 WatchSource:0}: Error finding container 1d12c3480400cddedaaa11868e36eff35ebecbc483e8b37c39037ecd588cbc61: Status 404 returned error can't find the container with id 1d12c3480400cddedaaa11868e36eff35ebecbc483e8b37c39037ecd588cbc61
Jul 28 01:33:36.098060 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:36.098026    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-sc9gn" event={"ID":"9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3","Type":"ContainerStarted","Data":"1d12c3480400cddedaaa11868e36eff35ebecbc483e8b37c39037ecd588cbc61"}
Jul 28 01:33:36.099109 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:36.099078    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-tvgjt" event={"ID":"0d3eac21-7291-4d7e-8209-4d55b8a88f22","Type":"ContainerStarted","Data":"f5e22080f2829a62bc829147a71b84f6b7bdee5e0f9efd79e460b6eee0c2aad2"}
Jul 28 01:33:40.114532 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:40.114480    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-sc9gn" event={"ID":"9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3","Type":"ContainerStarted","Data":"912f10a1714efdf19c060e7729483413fbe812440c2fad58e8a4e16515a14dc0"}
Jul 28 01:33:40.115013 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:40.114603    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-sc9gn"
Jul 28 01:33:40.116028 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:40.116004    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-tvgjt" event={"ID":"0d3eac21-7291-4d7e-8209-4d55b8a88f22","Type":"ContainerStarted","Data":"3746e1ec79ca1f69ed3896fe95d46ed05aa4c892f88631d6b11bc1c3a630b6c7"}
Jul 28 01:33:40.116107 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:40.116052    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-tvgjt"
Jul 28 01:33:40.130241 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:40.130201    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-sc9gn" podStartSLOduration=2.177731028 podStartE2EDuration="6.130188945s" totalImagesPullingTime="3.952457917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:33:40.129327671 +0000 UTC m=+467.138846107" watchObservedRunningTime="2026-07-28 01:33:40.130188945 +0000 UTC m=+467.139707368"
Jul 28 01:33:40.142911 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:40.142875    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-tvgjt" podStartSLOduration=1.548913733 podStartE2EDuration="6.142862518s" totalImagesPullingTime="4.593948785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:33:40.141891648 +0000 UTC m=+467.151410091" watchObservedRunningTime="2026-07-28 01:33:40.142862518 +0000 UTC m=+467.152380941"
Jul 28 01:33:51.121054 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:51.121026    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-tvgjt"
Jul 28 01:33:51.122971 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:33:51.122952    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-sc9gn"
Jul 28 01:35:58.020080 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.020048    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"]
Jul 28 01:35:58.022554 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.022537    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.025234 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.025199    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 28 01:35:58.025234 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.025228    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 28 01:35:58.025393 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.025241    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 01:35:58.025393 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.025233    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 01:35:58.025393 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.025286    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 01:35:58.025393 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.025300    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 01:35:58.025549 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.025445    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 01:35:58.033560 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.033517    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"]
Jul 28 01:35:58.084605 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.084580    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.084698 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.084614    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78fe8ba7-6e93-4504-b1b2-1d53357269c6-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.084698 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.084636    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frj4l\" (UniqueName: \"kubernetes.io/projected/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kube-api-access-frj4l\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.084698 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.084658    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.084852 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.084702    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78fe8ba7-6e93-4504-b1b2-1d53357269c6-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.185862 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.185843    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78fe8ba7-6e93-4504-b1b2-1d53357269c6-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.185969 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.185871    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frj4l\" (UniqueName: \"kubernetes.io/projected/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kube-api-access-frj4l\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.185969 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.185890    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.185969 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.185910    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78fe8ba7-6e93-4504-b1b2-1d53357269c6-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.186185 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.186166    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.186331 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.186283    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.186639 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.186606    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78fe8ba7-6e93-4504-b1b2-1d53357269c6-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.186871 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.186734    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.188549 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.188512    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78fe8ba7-6e93-4504-b1b2-1d53357269c6-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.194628 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.194610    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frj4l\" (UniqueName: \"kubernetes.io/projected/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kube-api-access-frj4l\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.332033 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.331977    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:35:58.462297 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.462276    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"]
Jul 28 01:35:58.465027 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:35:58.464997    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78fe8ba7_6e93_4504_b1b2_1d53357269c6.slice/crio-42603e1d17f7b19ee68d69749122fa21456f0f2f6752e4d0cf3fc8b62d5d3c07 WatchSource:0}: Error finding container 42603e1d17f7b19ee68d69749122fa21456f0f2f6752e4d0cf3fc8b62d5d3c07: Status 404 returned error can't find the container with id 42603e1d17f7b19ee68d69749122fa21456f0f2f6752e4d0cf3fc8b62d5d3c07
Jul 28 01:35:58.541759 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:35:58.541731    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" event={"ID":"78fe8ba7-6e93-4504-b1b2-1d53357269c6","Type":"ContainerStarted","Data":"42603e1d17f7b19ee68d69749122fa21456f0f2f6752e4d0cf3fc8b62d5d3c07"}
Jul 28 01:36:02.558302 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:02.558262    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" event={"ID":"78fe8ba7-6e93-4504-b1b2-1d53357269c6","Type":"ContainerStarted","Data":"5708cc4e1b93ed930cb8023cfee749b30730bcd61dfdaf4c63a99f0181cf1032"}
Jul 28 01:36:06.573782 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:06.573750    2555 generic.go:356] "Generic (PLEG): container finished" podID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerID="5708cc4e1b93ed930cb8023cfee749b30730bcd61dfdaf4c63a99f0181cf1032" exitCode=0
Jul 28 01:36:06.574133 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:06.573826    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" event={"ID":"78fe8ba7-6e93-4504-b1b2-1d53357269c6","Type":"ContainerDied","Data":"5708cc4e1b93ed930cb8023cfee749b30730bcd61dfdaf4c63a99f0181cf1032"}
Jul 28 01:36:19.631643 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:19.631609    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" event={"ID":"78fe8ba7-6e93-4504-b1b2-1d53357269c6","Type":"ContainerStarted","Data":"fda39f75f488e294fbccbc1aa881a2352d211ec1be43e8ff43795bce8fc7ffd4"}
Jul 28 01:36:22.644512 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:22.644476    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" event={"ID":"78fe8ba7-6e93-4504-b1b2-1d53357269c6","Type":"ContainerStarted","Data":"a4c6e7089b371c232819242b82d55f2d9b294cf3a28358d4d761a81236ac9d59"}
Jul 28 01:36:24.653361 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:24.653327    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" event={"ID":"78fe8ba7-6e93-4504-b1b2-1d53357269c6","Type":"ContainerStarted","Data":"163a79f695977f56eec648caee7a669adaa910d344d7bb5dccdddf2671d6bedf"}
Jul 28 01:36:24.653829 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:24.653553    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:36:24.653829 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:24.653598    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:36:24.654785 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:24.654762    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 01:36:24.672851 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:24.672806    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podStartSLOduration=2.093003811 podStartE2EDuration="26.6727933s" totalImagesPullingTime="20.82517349s" totalInitContainerRuntime="3.754615999s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:35:58 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:36:24.67232219 +0000 UTC m=+631.681840614" watchObservedRunningTime="2026-07-28 01:36:24.6727933 +0000 UTC m=+631.682311723"
Jul 28 01:36:25.656209 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:25.656163    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 01:36:25.656587 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:25.656237    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:36:25.657460 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:25.657432    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:36:26.660011 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:26.659968    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 01:36:26.660497 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:26.660473    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:36:26.663933 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:26.663914    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:36:27.664006 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:27.663964    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 01:36:27.664378 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:27.664260    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:36:37.664382 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:37.664330    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 01:36:37.665022 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:37.664735    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:36:47.664231 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:47.664183    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 01:36:47.664676 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:47.664647    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:36:57.663813 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:57.663767    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 01:36:57.664263 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:36:57.664234    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:07.664616 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:07.664570    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 01:37:07.665153 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:07.665131    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:17.663813 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:17.663768    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 01:37:17.664281 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:17.664148    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:27.664471 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:27.664444    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:37:27.664869 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:27.664596    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:37:43.203030 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.203002    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"]
Jul 28 01:37:43.203381 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.203248    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kserve-container" containerID="cri-o://fda39f75f488e294fbccbc1aa881a2352d211ec1be43e8ff43795bce8fc7ffd4" gracePeriod=30
Jul 28 01:37:43.203381 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.203341    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="agent" containerID="cri-o://163a79f695977f56eec648caee7a669adaa910d344d7bb5dccdddf2671d6bedf" gracePeriod=30
Jul 28 01:37:43.203486 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.203372    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kube-rbac-proxy" containerID="cri-o://a4c6e7089b371c232819242b82d55f2d9b294cf3a28358d4d761a81236ac9d59" gracePeriod=30
Jul 28 01:37:43.290103 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.290076    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"]
Jul 28 01:37:43.292489 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.292471    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.295192 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.295177    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 28 01:37:43.295445 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.295431    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 28 01:37:43.306770 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.306746    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40393776-09b8-4e98-812f-5723c4d55a0f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.306852 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.306791    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40393776-09b8-4e98-812f-5723c4d55a0f-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.306852 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.306834    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40393776-09b8-4e98-812f-5723c4d55a0f-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.306922 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.306862    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40393776-09b8-4e98-812f-5723c4d55a0f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.306922 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.306880    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcg42\" (UniqueName: \"kubernetes.io/projected/40393776-09b8-4e98-812f-5723c4d55a0f-kube-api-access-dcg42\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.310664 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.310642    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"]
Jul 28 01:37:43.408105 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.408084    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40393776-09b8-4e98-812f-5723c4d55a0f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.408218 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.408125    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40393776-09b8-4e98-812f-5723c4d55a0f-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.408218 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.408146    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40393776-09b8-4e98-812f-5723c4d55a0f-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.408330 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:37:43.408246    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 28 01:37:43.408330 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.408280    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40393776-09b8-4e98-812f-5723c4d55a0f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.408330 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:37:43.408320    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40393776-09b8-4e98-812f-5723c4d55a0f-proxy-tls podName:40393776-09b8-4e98-812f-5723c4d55a0f nodeName:}" failed. No retries permitted until 2026-07-28 01:37:43.908298499 +0000 UTC m=+710.917816902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/40393776-09b8-4e98-812f-5723c4d55a0f-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" (UID: "40393776-09b8-4e98-812f-5723c4d55a0f") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 28 01:37:43.408486 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.408360    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcg42\" (UniqueName: \"kubernetes.io/projected/40393776-09b8-4e98-812f-5723c4d55a0f-kube-api-access-dcg42\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.408572 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.408509    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40393776-09b8-4e98-812f-5723c4d55a0f-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.408764 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.408746    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40393776-09b8-4e98-812f-5723c4d55a0f-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.408883 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.408862    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40393776-09b8-4e98-812f-5723c4d55a0f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.416859 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.416837    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcg42\" (UniqueName: \"kubernetes.io/projected/40393776-09b8-4e98-812f-5723c4d55a0f-kube-api-access-dcg42\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.911914 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.911878    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40393776-09b8-4e98-812f-5723c4d55a0f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.914404 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.914383    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40393776-09b8-4e98-812f-5723c4d55a0f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:43.919628 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.919605    2555 generic.go:356] "Generic (PLEG): container finished" podID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerID="a4c6e7089b371c232819242b82d55f2d9b294cf3a28358d4d761a81236ac9d59" exitCode=2
Jul 28 01:37:43.919720 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:43.919650    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" event={"ID":"78fe8ba7-6e93-4504-b1b2-1d53357269c6","Type":"ContainerDied","Data":"a4c6e7089b371c232819242b82d55f2d9b294cf3a28358d4d761a81236ac9d59"}
Jul 28 01:37:44.202249 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:44.202168    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:44.326140 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:44.326108    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"]
Jul 28 01:37:44.328989 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:37:44.328962    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40393776_09b8_4e98_812f_5723c4d55a0f.slice/crio-71423a12fc4062e9a3a6f91fba191c0d15636b23f93503af296c5d6593f1fc32 WatchSource:0}: Error finding container 71423a12fc4062e9a3a6f91fba191c0d15636b23f93503af296c5d6593f1fc32: Status 404 returned error can't find the container with id 71423a12fc4062e9a3a6f91fba191c0d15636b23f93503af296c5d6593f1fc32
Jul 28 01:37:44.924316 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:44.924282    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" event={"ID":"40393776-09b8-4e98-812f-5723c4d55a0f","Type":"ContainerStarted","Data":"d0af0a445e189ad4d835879722891ca43c8f6d43d55711a4b65a2f4e697170d0"}
Jul 28 01:37:44.924496 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:44.924352    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" event={"ID":"40393776-09b8-4e98-812f-5723c4d55a0f","Type":"ContainerStarted","Data":"71423a12fc4062e9a3a6f91fba191c0d15636b23f93503af296c5d6593f1fc32"}
Jul 28 01:37:46.660208 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:46.660167    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused"
Jul 28 01:37:47.664090 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:47.663999    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 01:37:47.664481 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:47.664346    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:47.936115 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:47.936028    2555 generic.go:356] "Generic (PLEG): container finished" podID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerID="fda39f75f488e294fbccbc1aa881a2352d211ec1be43e8ff43795bce8fc7ffd4" exitCode=0
Jul 28 01:37:47.936115 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:47.936090    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" event={"ID":"78fe8ba7-6e93-4504-b1b2-1d53357269c6","Type":"ContainerDied","Data":"fda39f75f488e294fbccbc1aa881a2352d211ec1be43e8ff43795bce8fc7ffd4"}
Jul 28 01:37:48.940501 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:48.940471    2555 generic.go:356] "Generic (PLEG): container finished" podID="40393776-09b8-4e98-812f-5723c4d55a0f" containerID="d0af0a445e189ad4d835879722891ca43c8f6d43d55711a4b65a2f4e697170d0" exitCode=0
Jul 28 01:37:48.940948 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:48.940543    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" event={"ID":"40393776-09b8-4e98-812f-5723c4d55a0f","Type":"ContainerDied","Data":"d0af0a445e189ad4d835879722891ca43c8f6d43d55711a4b65a2f4e697170d0"}
Jul 28 01:37:49.946811 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:49.946775    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" event={"ID":"40393776-09b8-4e98-812f-5723c4d55a0f","Type":"ContainerStarted","Data":"9d2b80fcb4de79627b260d4f5b6804753c0dc4d8b5ecbecf87badb82516105c4"}
Jul 28 01:37:49.947188 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:49.946865    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" event={"ID":"40393776-09b8-4e98-812f-5723c4d55a0f","Type":"ContainerStarted","Data":"8abadac5cea64c867197bee1296c2c23c76fc91a761d2eef454c64391b2bf3f3"}
Jul 28 01:37:49.947188 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:49.946889    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" event={"ID":"40393776-09b8-4e98-812f-5723c4d55a0f","Type":"ContainerStarted","Data":"dab691c2e17ac09767f6c75771d0548d0040d4157d938f271539703c49f2b842"}
Jul 28 01:37:49.947297 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:49.947217    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:49.947373 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:49.947354    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:49.948509 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:49.948481    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:5000: connect: connection refused"
Jul 28 01:37:49.969807 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:49.969748    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podStartSLOduration=3.370653017 podStartE2EDuration="6.969737014s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.599083997s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:37:49.968698361 +0000 UTC m=+716.978216809" watchObservedRunningTime="2026-07-28 01:37:49.969737014 +0000 UTC m=+716.979255437"
Jul 28 01:37:50.950592 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:50.950550    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:5000: connect: connection refused"
Jul 28 01:37:50.950979 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:50.950635    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:50.951670 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:50.951644    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:51.660795 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:51.660756    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused"
Jul 28 01:37:51.953992 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:51.953897    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:5000: connect: connection refused"
Jul 28 01:37:51.954385 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:51.954363    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:56.660354 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:56.660313    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused"
Jul 28 01:37:56.660792 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:56.660433    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:37:56.958706 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:56.958618    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:37:56.959196 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:56.959158    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:5000: connect: connection refused"
Jul 28 01:37:56.959475 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:56.959450    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:57.664024 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:57.663979    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 01:37:57.664390 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:37:57.664320    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:38:01.660833 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:01.660791    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused"
Jul 28 01:38:06.660839 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:06.660800    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused"
Jul 28 01:38:06.959830 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:06.959742    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:5000: connect: connection refused"
Jul 28 01:38:06.960243 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:06.960220    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:38:07.664027 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:07.663980    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 28 01:38:07.664409 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:07.664120    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:38:07.664409 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:07.664342    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:38:07.664500 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:07.664430    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:38:11.660775 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:11.660741    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused"
Jul 28 01:38:13.344613 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:13.344591    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:38:13.420449 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:13.420420    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kserve-provision-location\") pod \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") "
Jul 28 01:38:13.420599 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:13.420482    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frj4l\" (UniqueName: \"kubernetes.io/projected/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kube-api-access-frj4l\") pod \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") "
Jul 28 01:38:13.420599 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:13.420514    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78fe8ba7-6e93-4504-b1b2-1d53357269c6-cabundle-cert\") pod \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") "
Jul 28 01:38:13.420599 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:13.420559    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kube-rbac-proxy-sar-config\") pod \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") "
Jul 28 01:38:13.420742 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:13.420623    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78fe8ba7-6e93-4504-b1b2-1d53357269c6-proxy-tls\") pod \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\" (UID: \"78fe8ba7-6e93-4504-b1b2-1d53357269c6\") "
Jul 28 01:38:13.420742 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:13.420628    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "78fe8ba7-6e93-4504-b1b2-1d53357269c6" (UID: "78fe8ba7-6e93-4504-b1b2-1d53357269c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:38:13.420867 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:13.420821    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78fe8ba7-6e93-4504-b1b2-1d53357269c6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "78fe8ba7-6e93-4504-b1b2-1d53357269c6" (UID: "78fe8ba7-6e93-4504-b1b2-1d53357269c6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:38:13.420938 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:13.420922    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:38:13.420990 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:13.420941    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78fe8ba7-6e93-4504-b1b2-1d53357269c6-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:38:13.421038 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:13.420997    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "78fe8ba7-6e93-4504-b1b2-1d53357269c6" (UID: "78fe8ba7-6e93-4504-b1b2-1d53357269c6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:38:13.422726 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:13.422704    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kube-api-access-frj4l" (OuterVolumeSpecName: "kube-api-access-frj4l") pod "78fe8ba7-6e93-4504-b1b2-1d53357269c6" (UID: "78fe8ba7-6e93-4504-b1b2-1d53357269c6"). InnerVolumeSpecName "kube-api-access-frj4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:38:13.422800 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:13.422777    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78fe8ba7-6e93-4504-b1b2-1d53357269c6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "78fe8ba7-6e93-4504-b1b2-1d53357269c6" (UID: "78fe8ba7-6e93-4504-b1b2-1d53357269c6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:38:13.521316 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:13.521292    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:38:13.521316 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:13.521316    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78fe8ba7-6e93-4504-b1b2-1d53357269c6-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:38:13.521474 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:13.521327    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frj4l\" (UniqueName: \"kubernetes.io/projected/78fe8ba7-6e93-4504-b1b2-1d53357269c6-kube-api-access-frj4l\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:38:14.029150 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.029114    2555 generic.go:356] "Generic (PLEG): container finished" podID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerID="163a79f695977f56eec648caee7a669adaa910d344d7bb5dccdddf2671d6bedf" exitCode=0
Jul 28 01:38:14.029343 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.029172    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" event={"ID":"78fe8ba7-6e93-4504-b1b2-1d53357269c6","Type":"ContainerDied","Data":"163a79f695977f56eec648caee7a669adaa910d344d7bb5dccdddf2671d6bedf"}
Jul 28 01:38:14.029343 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.029180    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"
Jul 28 01:38:14.029343 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.029224    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq" event={"ID":"78fe8ba7-6e93-4504-b1b2-1d53357269c6","Type":"ContainerDied","Data":"42603e1d17f7b19ee68d69749122fa21456f0f2f6752e4d0cf3fc8b62d5d3c07"}
Jul 28 01:38:14.029343 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.029253    2555 scope.go:117] "RemoveContainer" containerID="163a79f695977f56eec648caee7a669adaa910d344d7bb5dccdddf2671d6bedf"
Jul 28 01:38:14.036753 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.036725    2555 scope.go:117] "RemoveContainer" containerID="a4c6e7089b371c232819242b82d55f2d9b294cf3a28358d4d761a81236ac9d59"
Jul 28 01:38:14.043436 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.043421    2555 scope.go:117] "RemoveContainer" containerID="fda39f75f488e294fbccbc1aa881a2352d211ec1be43e8ff43795bce8fc7ffd4"
Jul 28 01:38:14.049309 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.049290    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"]
Jul 28 01:38:14.050944 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.050929    2555 scope.go:117] "RemoveContainer" containerID="5708cc4e1b93ed930cb8023cfee749b30730bcd61dfdaf4c63a99f0181cf1032"
Jul 28 01:38:14.053062 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.053043    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-z7gqq"]
Jul 28 01:38:14.057899 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.057884    2555 scope.go:117] "RemoveContainer" containerID="163a79f695977f56eec648caee7a669adaa910d344d7bb5dccdddf2671d6bedf"
Jul 28 01:38:14.058155 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:38:14.058136    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163a79f695977f56eec648caee7a669adaa910d344d7bb5dccdddf2671d6bedf\": container with ID starting with 163a79f695977f56eec648caee7a669adaa910d344d7bb5dccdddf2671d6bedf not found: ID does not exist" containerID="163a79f695977f56eec648caee7a669adaa910d344d7bb5dccdddf2671d6bedf"
Jul 28 01:38:14.058224 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.058167    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163a79f695977f56eec648caee7a669adaa910d344d7bb5dccdddf2671d6bedf"} err="failed to get container status \"163a79f695977f56eec648caee7a669adaa910d344d7bb5dccdddf2671d6bedf\": rpc error: code = NotFound desc = could not find container \"163a79f695977f56eec648caee7a669adaa910d344d7bb5dccdddf2671d6bedf\": container with ID starting with 163a79f695977f56eec648caee7a669adaa910d344d7bb5dccdddf2671d6bedf not found: ID does not exist"
Jul 28 01:38:14.058224 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.058190    2555 scope.go:117] "RemoveContainer" containerID="a4c6e7089b371c232819242b82d55f2d9b294cf3a28358d4d761a81236ac9d59"
Jul 28 01:38:14.058402 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:38:14.058385    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c6e7089b371c232819242b82d55f2d9b294cf3a28358d4d761a81236ac9d59\": container with ID starting with a4c6e7089b371c232819242b82d55f2d9b294cf3a28358d4d761a81236ac9d59 not found: ID does not exist" containerID="a4c6e7089b371c232819242b82d55f2d9b294cf3a28358d4d761a81236ac9d59"
Jul 28 01:38:14.058447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.058412    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c6e7089b371c232819242b82d55f2d9b294cf3a28358d4d761a81236ac9d59"} err="failed to get container status \"a4c6e7089b371c232819242b82d55f2d9b294cf3a28358d4d761a81236ac9d59\": rpc error: code = NotFound desc = could not find container \"a4c6e7089b371c232819242b82d55f2d9b294cf3a28358d4d761a81236ac9d59\": container with ID starting with a4c6e7089b371c232819242b82d55f2d9b294cf3a28358d4d761a81236ac9d59 not found: ID does not exist"
Jul 28 01:38:14.058447 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.058435    2555 scope.go:117] "RemoveContainer" containerID="fda39f75f488e294fbccbc1aa881a2352d211ec1be43e8ff43795bce8fc7ffd4"
Jul 28 01:38:14.058696 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:38:14.058674    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda39f75f488e294fbccbc1aa881a2352d211ec1be43e8ff43795bce8fc7ffd4\": container with ID starting with fda39f75f488e294fbccbc1aa881a2352d211ec1be43e8ff43795bce8fc7ffd4 not found: ID does not exist" containerID="fda39f75f488e294fbccbc1aa881a2352d211ec1be43e8ff43795bce8fc7ffd4"
Jul 28 01:38:14.058747 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.058705    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda39f75f488e294fbccbc1aa881a2352d211ec1be43e8ff43795bce8fc7ffd4"} err="failed to get container status \"fda39f75f488e294fbccbc1aa881a2352d211ec1be43e8ff43795bce8fc7ffd4\": rpc error: code = NotFound desc = could not find container \"fda39f75f488e294fbccbc1aa881a2352d211ec1be43e8ff43795bce8fc7ffd4\": container with ID starting with fda39f75f488e294fbccbc1aa881a2352d211ec1be43e8ff43795bce8fc7ffd4 not found: ID does not exist"
Jul 28 01:38:14.058747 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.058726    2555 scope.go:117] "RemoveContainer" containerID="5708cc4e1b93ed930cb8023cfee749b30730bcd61dfdaf4c63a99f0181cf1032"
Jul 28 01:38:14.059017 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:38:14.059001    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5708cc4e1b93ed930cb8023cfee749b30730bcd61dfdaf4c63a99f0181cf1032\": container with ID starting with 5708cc4e1b93ed930cb8023cfee749b30730bcd61dfdaf4c63a99f0181cf1032 not found: ID does not exist" containerID="5708cc4e1b93ed930cb8023cfee749b30730bcd61dfdaf4c63a99f0181cf1032"
Jul 28 01:38:14.059068 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:14.059021    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5708cc4e1b93ed930cb8023cfee749b30730bcd61dfdaf4c63a99f0181cf1032"} err="failed to get container status \"5708cc4e1b93ed930cb8023cfee749b30730bcd61dfdaf4c63a99f0181cf1032\": rpc error: code = NotFound desc = could not find container \"5708cc4e1b93ed930cb8023cfee749b30730bcd61dfdaf4c63a99f0181cf1032\": container with ID starting with 5708cc4e1b93ed930cb8023cfee749b30730bcd61dfdaf4c63a99f0181cf1032 not found: ID does not exist"
Jul 28 01:38:15.451431 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:15.451391    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" path="/var/lib/kubelet/pods/78fe8ba7-6e93-4504-b1b2-1d53357269c6/volumes"
Jul 28 01:38:16.959391 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:16.959343    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:5000: connect: connection refused"
Jul 28 01:38:16.959840 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:16.959816    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:38:26.959950 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:26.959898    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:5000: connect: connection refused"
Jul 28 01:38:26.960453 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:26.960340    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:38:36.959432 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:36.959381    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:5000: connect: connection refused"
Jul 28 01:38:36.959875 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:36.959853    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:38:46.959359 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:46.959311    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:5000: connect: connection refused"
Jul 28 01:38:46.959789 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:46.959763    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:38:56.960158 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:56.960130    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:38:56.960576 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:38:56.960401    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:39:08.342369 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.342337    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"]
Jul 28 01:39:08.342837 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.342616    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kserve-container"
Jul 28 01:39:08.342837 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.342629    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kserve-container"
Jul 28 01:39:08.342837 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.342637    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="storage-initializer"
Jul 28 01:39:08.342837 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.342643    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="storage-initializer"
Jul 28 01:39:08.342837 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.342654    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kube-rbac-proxy"
Jul 28 01:39:08.342837 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.342659    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kube-rbac-proxy"
Jul 28 01:39:08.342837 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.342670    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="agent"
Jul 28 01:39:08.342837 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.342675    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="agent"
Jul 28 01:39:08.342837 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.342720    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kserve-container"
Jul 28 01:39:08.342837 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.342726    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="kube-rbac-proxy"
Jul 28 01:39:08.342837 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.342733    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="78fe8ba7-6e93-4504-b1b2-1d53357269c6" containerName="agent"
Jul 28 01:39:08.346692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.346675    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:39:08.349281 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.349262    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 28 01:39:08.349802 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.349782    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 28 01:39:08.354612 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.354575    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"]
Jul 28 01:39:08.385538 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.385494    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"]
Jul 28 01:39:08.385759 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.385729    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kserve-container" containerID="cri-o://dab691c2e17ac09767f6c75771d0548d0040d4157d938f271539703c49f2b842" gracePeriod=30
Jul 28 01:39:08.385872 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.385764    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="agent" containerID="cri-o://9d2b80fcb4de79627b260d4f5b6804753c0dc4d8b5ecbecf87badb82516105c4" gracePeriod=30
Jul 28 01:39:08.385872 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.385768    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kube-rbac-proxy" containerID="cri-o://8abadac5cea64c867197bee1296c2c23c76fc91a761d2eef454c64391b2bf3f3" gracePeriod=30
Jul 28 01:39:08.484290 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.484266    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcq9z\" (UniqueName: \"kubernetes.io/projected/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-kube-api-access-kcq9z\") pod \"message-dumper-predictor-6575bcfdfd-6zv2z\" (UID: \"b7b00ab5-1811-45ca-8f56-ed6e19ec551f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:39:08.484421 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.484304    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-6zv2z\" (UID: \"b7b00ab5-1811-45ca-8f56-ed6e19ec551f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:39:08.484421 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.484326    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-6zv2z\" (UID: \"b7b00ab5-1811-45ca-8f56-ed6e19ec551f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:39:08.585006 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.584979    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcq9z\" (UniqueName: \"kubernetes.io/projected/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-kube-api-access-kcq9z\") pod \"message-dumper-predictor-6575bcfdfd-6zv2z\" (UID: \"b7b00ab5-1811-45ca-8f56-ed6e19ec551f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:39:08.585116 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.585028    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-6zv2z\" (UID: \"b7b00ab5-1811-45ca-8f56-ed6e19ec551f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:39:08.585116 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.585065    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-6zv2z\" (UID: \"b7b00ab5-1811-45ca-8f56-ed6e19ec551f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:39:08.585724 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.585702    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-6zv2z\" (UID: \"b7b00ab5-1811-45ca-8f56-ed6e19ec551f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:39:08.587251 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.587233    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-6zv2z\" (UID: \"b7b00ab5-1811-45ca-8f56-ed6e19ec551f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:39:08.593513 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.593465    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcq9z\" (UniqueName: \"kubernetes.io/projected/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-kube-api-access-kcq9z\") pod \"message-dumper-predictor-6575bcfdfd-6zv2z\" (UID: \"b7b00ab5-1811-45ca-8f56-ed6e19ec551f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:39:08.656442 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.656407    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:39:08.782725 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.782696    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"]
Jul 28 01:39:08.785486 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:39:08.785459    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b00ab5_1811_45ca_8f56_ed6e19ec551f.slice/crio-64779e999d0edc48d8b53cf22c7defafd9bb600e495e0d5db143d35113ac0ba6 WatchSource:0}: Error finding container 64779e999d0edc48d8b53cf22c7defafd9bb600e495e0d5db143d35113ac0ba6: Status 404 returned error can't find the container with id 64779e999d0edc48d8b53cf22c7defafd9bb600e495e0d5db143d35113ac0ba6
Jul 28 01:39:08.787238 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:08.787224    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:39:09.217345 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:09.217311    2555 generic.go:356] "Generic (PLEG): container finished" podID="40393776-09b8-4e98-812f-5723c4d55a0f" containerID="8abadac5cea64c867197bee1296c2c23c76fc91a761d2eef454c64391b2bf3f3" exitCode=2
Jul 28 01:39:09.217515 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:09.217374    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" event={"ID":"40393776-09b8-4e98-812f-5723c4d55a0f","Type":"ContainerDied","Data":"8abadac5cea64c867197bee1296c2c23c76fc91a761d2eef454c64391b2bf3f3"}
Jul 28 01:39:09.218414 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:09.218395    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z" event={"ID":"b7b00ab5-1811-45ca-8f56-ed6e19ec551f","Type":"ContainerStarted","Data":"64779e999d0edc48d8b53cf22c7defafd9bb600e495e0d5db143d35113ac0ba6"}
Jul 28 01:39:10.223297 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:10.223267    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z" event={"ID":"b7b00ab5-1811-45ca-8f56-ed6e19ec551f","Type":"ContainerStarted","Data":"61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4"}
Jul 28 01:39:10.223691 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:10.223320    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z" event={"ID":"b7b00ab5-1811-45ca-8f56-ed6e19ec551f","Type":"ContainerStarted","Data":"03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46"}
Jul 28 01:39:10.223691 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:10.223422    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:39:10.241464 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:10.241416    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z" podStartSLOduration=1.008816546 podStartE2EDuration="2.24140011s" totalImagesPullingTime="1.232583564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:39:10.239882665 +0000 UTC m=+797.249401126" watchObservedRunningTime="2026-07-28 01:39:10.24140011 +0000 UTC m=+797.250918534"
Jul 28 01:39:11.227358 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:11.227323    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:39:11.228743 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:11.228718    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:39:11.954720 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:11.954681    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 28 01:39:13.236322 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:13.236288    2555 generic.go:356] "Generic (PLEG): container finished" podID="40393776-09b8-4e98-812f-5723c4d55a0f" containerID="dab691c2e17ac09767f6c75771d0548d0040d4157d938f271539703c49f2b842" exitCode=0
Jul 28 01:39:13.236694 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:13.236361    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" event={"ID":"40393776-09b8-4e98-812f-5723c4d55a0f","Type":"ContainerDied","Data":"dab691c2e17ac09767f6c75771d0548d0040d4157d938f271539703c49f2b842"}
Jul 28 01:39:16.955077 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:16.954996    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 28 01:39:16.959287 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:16.959262    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:5000: connect: connection refused"
Jul 28 01:39:16.959719 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:16.959697    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:39:18.241266 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:18.241230    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:39:21.954496 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:21.954459    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 28 01:39:21.954920 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:21.954656    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:39:26.955115 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:26.955073    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 28 01:39:26.959427 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:26.959401    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:5000: connect: connection refused"
Jul 28 01:39:26.959776 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:26.959751    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:39:28.411649 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.411620    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"]
Jul 28 01:39:28.414681 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.414664    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:28.422156 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.422136    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 28 01:39:28.422263 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.422164    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 28 01:39:28.425023 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.425004    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"]
Jul 28 01:39:28.510576 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.510554    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-h8b5t\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:28.510697 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.510602    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-h8b5t\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:28.510697 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.510629    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfrxb\" (UniqueName: \"kubernetes.io/projected/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kube-api-access-qfrxb\") pod \"isvc-logger-predictor-6b6746bb7-h8b5t\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:28.510784 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.510714    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-h8b5t\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:28.510784 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.510751    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-h8b5t\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:28.611899 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.611875    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-h8b5t\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:28.612027 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.611922    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-h8b5t\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:28.612027 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.611944    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-h8b5t\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:28.612027 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.611967    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfrxb\" (UniqueName: \"kubernetes.io/projected/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kube-api-access-qfrxb\") pod \"isvc-logger-predictor-6b6746bb7-h8b5t\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:28.612179 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.612034    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-h8b5t\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:28.612179 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:39:28.612043    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 28 01:39:28.612179 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:39:28.612113    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-proxy-tls podName:05b64515-c5d5-42fe-97bb-5b16b6a48e4e nodeName:}" failed. No retries permitted until 2026-07-28 01:39:29.112093464 +0000 UTC m=+816.121611867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-proxy-tls") pod "isvc-logger-predictor-6b6746bb7-h8b5t" (UID: "05b64515-c5d5-42fe-97bb-5b16b6a48e4e") : secret "isvc-logger-predictor-serving-cert" not found
Jul 28 01:39:28.612374 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.612354    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-h8b5t\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:28.612768 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.612748    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-h8b5t\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:28.612846 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.612753    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-h8b5t\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:28.620221 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:28.620201    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfrxb\" (UniqueName: \"kubernetes.io/projected/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kube-api-access-qfrxb\") pod \"isvc-logger-predictor-6b6746bb7-h8b5t\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:29.115545 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:29.115490    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-h8b5t\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:29.117913 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:29.117884    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-h8b5t\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:29.325014 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:29.324987    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:29.457301 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:29.457268    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"]
Jul 28 01:39:29.460076 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:39:29.460051    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b64515_c5d5_42fe_97bb_5b16b6a48e4e.slice/crio-4a0930406707d3e5b2f77de5f7af846f99fe9a0c2604c73b5daee99cc5adecb0 WatchSource:0}: Error finding container 4a0930406707d3e5b2f77de5f7af846f99fe9a0c2604c73b5daee99cc5adecb0: Status 404 returned error can't find the container with id 4a0930406707d3e5b2f77de5f7af846f99fe9a0c2604c73b5daee99cc5adecb0
Jul 28 01:39:30.297026 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:30.296992    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" event={"ID":"05b64515-c5d5-42fe-97bb-5b16b6a48e4e","Type":"ContainerStarted","Data":"1b5e6d7597609d1b4680f8253aee4a3ef022487268075d93afcd392d08db9322"}
Jul 28 01:39:30.297191 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:30.297055    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" event={"ID":"05b64515-c5d5-42fe-97bb-5b16b6a48e4e","Type":"ContainerStarted","Data":"4a0930406707d3e5b2f77de5f7af846f99fe9a0c2604c73b5daee99cc5adecb0"}
Jul 28 01:39:31.955005 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:31.954970    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 28 01:39:33.307935 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:33.307905    2555 generic.go:356] "Generic (PLEG): container finished" podID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerID="1b5e6d7597609d1b4680f8253aee4a3ef022487268075d93afcd392d08db9322" exitCode=0
Jul 28 01:39:33.308320 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:33.307967    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" event={"ID":"05b64515-c5d5-42fe-97bb-5b16b6a48e4e","Type":"ContainerDied","Data":"1b5e6d7597609d1b4680f8253aee4a3ef022487268075d93afcd392d08db9322"}
Jul 28 01:39:34.319238 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:34.319198    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" event={"ID":"05b64515-c5d5-42fe-97bb-5b16b6a48e4e","Type":"ContainerStarted","Data":"e264c7ea75bc7db4e12a4eab2bb233c558af142b0633886c9804a4d6a936cd39"}
Jul 28 01:39:34.319653 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:34.319294    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" event={"ID":"05b64515-c5d5-42fe-97bb-5b16b6a48e4e","Type":"ContainerStarted","Data":"7f4b5470a5814e27ec30df1cddf0f9443c654f36893e42bbe391f2aa6062a93a"}
Jul 28 01:39:34.319653 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:34.319321    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" event={"ID":"05b64515-c5d5-42fe-97bb-5b16b6a48e4e","Type":"ContainerStarted","Data":"6cb7683fee8dd17c725f45618ea450c93807493f3e62c3361d7151b2b8a3c1d1"}
Jul 28 01:39:34.319653 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:34.319626    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:34.319796 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:34.319673    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:34.319796 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:34.319696    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:34.321076 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:34.321044    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 01:39:34.321709 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:34.321689    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:39:34.340831 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:34.340782    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podStartSLOduration=2.676487127 podStartE2EDuration="6.34076415s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.664277023s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:39:34.338456137 +0000 UTC m=+821.347974581" watchObservedRunningTime="2026-07-28 01:39:34.34076415 +0000 UTC m=+821.350282640"
Jul 28 01:39:35.323005 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:35.322965    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 01:39:35.323489 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:35.323466    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:39:36.954693 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:36.954658    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 28 01:39:36.960090 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:36.960059    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:5000: connect: connection refused"
Jul 28 01:39:36.960216 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:36.960189    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:39:36.960280 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:36.960262    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:39:36.960366 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:36.960354    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:39:38.526444 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:38.526423    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:39:38.583688 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:38.583661    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcg42\" (UniqueName: \"kubernetes.io/projected/40393776-09b8-4e98-812f-5723c4d55a0f-kube-api-access-dcg42\") pod \"40393776-09b8-4e98-812f-5723c4d55a0f\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") "
Jul 28 01:39:38.583826 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:38.583711    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40393776-09b8-4e98-812f-5723c4d55a0f-kserve-provision-location\") pod \"40393776-09b8-4e98-812f-5723c4d55a0f\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") "
Jul 28 01:39:38.583826 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:38.583736    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40393776-09b8-4e98-812f-5723c4d55a0f-kube-rbac-proxy-sar-config\") pod \"40393776-09b8-4e98-812f-5723c4d55a0f\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") "
Jul 28 01:39:38.583938 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:38.583849    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40393776-09b8-4e98-812f-5723c4d55a0f-cabundle-cert\") pod \"40393776-09b8-4e98-812f-5723c4d55a0f\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") "
Jul 28 01:39:38.583938 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:38.583902    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40393776-09b8-4e98-812f-5723c4d55a0f-proxy-tls\") pod \"40393776-09b8-4e98-812f-5723c4d55a0f\" (UID: \"40393776-09b8-4e98-812f-5723c4d55a0f\") "
Jul 28 01:39:38.584038 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:38.583954    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40393776-09b8-4e98-812f-5723c4d55a0f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40393776-09b8-4e98-812f-5723c4d55a0f" (UID: "40393776-09b8-4e98-812f-5723c4d55a0f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:39:38.584132 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:38.584112    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40393776-09b8-4e98-812f-5723c4d55a0f-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:39:38.584185 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:38.584175    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40393776-09b8-4e98-812f-5723c4d55a0f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "40393776-09b8-4e98-812f-5723c4d55a0f" (UID: "40393776-09b8-4e98-812f-5723c4d55a0f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:39:38.584236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:38.584200    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40393776-09b8-4e98-812f-5723c4d55a0f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "40393776-09b8-4e98-812f-5723c4d55a0f" (UID: "40393776-09b8-4e98-812f-5723c4d55a0f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:39:38.585824 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:38.585802    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40393776-09b8-4e98-812f-5723c4d55a0f-kube-api-access-dcg42" (OuterVolumeSpecName: "kube-api-access-dcg42") pod "40393776-09b8-4e98-812f-5723c4d55a0f" (UID: "40393776-09b8-4e98-812f-5723c4d55a0f"). InnerVolumeSpecName "kube-api-access-dcg42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:39:38.585976 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:38.585957    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40393776-09b8-4e98-812f-5723c4d55a0f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40393776-09b8-4e98-812f-5723c4d55a0f" (UID: "40393776-09b8-4e98-812f-5723c4d55a0f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:39:38.684628 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:38.684575    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40393776-09b8-4e98-812f-5723c4d55a0f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:39:38.684628 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:38.684597    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40393776-09b8-4e98-812f-5723c4d55a0f-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:39:38.684628 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:38.684607    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40393776-09b8-4e98-812f-5723c4d55a0f-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:39:38.684628 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:38.684616    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcg42\" (UniqueName: \"kubernetes.io/projected/40393776-09b8-4e98-812f-5723c4d55a0f-kube-api-access-dcg42\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:39:39.342922 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.342888    2555 generic.go:356] "Generic (PLEG): container finished" podID="40393776-09b8-4e98-812f-5723c4d55a0f" containerID="9d2b80fcb4de79627b260d4f5b6804753c0dc4d8b5ecbecf87badb82516105c4" exitCode=0
Jul 28 01:39:39.343154 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.342929    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" event={"ID":"40393776-09b8-4e98-812f-5723c4d55a0f","Type":"ContainerDied","Data":"9d2b80fcb4de79627b260d4f5b6804753c0dc4d8b5ecbecf87badb82516105c4"}
Jul 28 01:39:39.343154 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.342951    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"
Jul 28 01:39:39.343154 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.342990    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq" event={"ID":"40393776-09b8-4e98-812f-5723c4d55a0f","Type":"ContainerDied","Data":"71423a12fc4062e9a3a6f91fba191c0d15636b23f93503af296c5d6593f1fc32"}
Jul 28 01:39:39.343154 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.343022    2555 scope.go:117] "RemoveContainer" containerID="9d2b80fcb4de79627b260d4f5b6804753c0dc4d8b5ecbecf87badb82516105c4"
Jul 28 01:39:39.351910 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.351892    2555 scope.go:117] "RemoveContainer" containerID="8abadac5cea64c867197bee1296c2c23c76fc91a761d2eef454c64391b2bf3f3"
Jul 28 01:39:39.358892 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.358877    2555 scope.go:117] "RemoveContainer" containerID="dab691c2e17ac09767f6c75771d0548d0040d4157d938f271539703c49f2b842"
Jul 28 01:39:39.363779 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.363760    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"]
Jul 28 01:39:39.366826 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.366717    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-q64dq"]
Jul 28 01:39:39.366886 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.366849    2555 scope.go:117] "RemoveContainer" containerID="d0af0a445e189ad4d835879722891ca43c8f6d43d55711a4b65a2f4e697170d0"
Jul 28 01:39:39.373489 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.373473    2555 scope.go:117] "RemoveContainer" containerID="9d2b80fcb4de79627b260d4f5b6804753c0dc4d8b5ecbecf87badb82516105c4"
Jul 28 01:39:39.373807 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:39:39.373789    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2b80fcb4de79627b260d4f5b6804753c0dc4d8b5ecbecf87badb82516105c4\": container with ID starting with 9d2b80fcb4de79627b260d4f5b6804753c0dc4d8b5ecbecf87badb82516105c4 not found: ID does not exist" containerID="9d2b80fcb4de79627b260d4f5b6804753c0dc4d8b5ecbecf87badb82516105c4"
Jul 28 01:39:39.373867 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.373814    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2b80fcb4de79627b260d4f5b6804753c0dc4d8b5ecbecf87badb82516105c4"} err="failed to get container status \"9d2b80fcb4de79627b260d4f5b6804753c0dc4d8b5ecbecf87badb82516105c4\": rpc error: code = NotFound desc = could not find container \"9d2b80fcb4de79627b260d4f5b6804753c0dc4d8b5ecbecf87badb82516105c4\": container with ID starting with 9d2b80fcb4de79627b260d4f5b6804753c0dc4d8b5ecbecf87badb82516105c4 not found: ID does not exist"
Jul 28 01:39:39.373867 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.373831    2555 scope.go:117] "RemoveContainer" containerID="8abadac5cea64c867197bee1296c2c23c76fc91a761d2eef454c64391b2bf3f3"
Jul 28 01:39:39.374080 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:39:39.374063    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8abadac5cea64c867197bee1296c2c23c76fc91a761d2eef454c64391b2bf3f3\": container with ID starting with 8abadac5cea64c867197bee1296c2c23c76fc91a761d2eef454c64391b2bf3f3 not found: ID does not exist" containerID="8abadac5cea64c867197bee1296c2c23c76fc91a761d2eef454c64391b2bf3f3"
Jul 28 01:39:39.374127 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.374087    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8abadac5cea64c867197bee1296c2c23c76fc91a761d2eef454c64391b2bf3f3"} err="failed to get container status \"8abadac5cea64c867197bee1296c2c23c76fc91a761d2eef454c64391b2bf3f3\": rpc error: code = NotFound desc = could not find container \"8abadac5cea64c867197bee1296c2c23c76fc91a761d2eef454c64391b2bf3f3\": container with ID starting with 8abadac5cea64c867197bee1296c2c23c76fc91a761d2eef454c64391b2bf3f3 not found: ID does not exist"
Jul 28 01:39:39.374127 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.374102    2555 scope.go:117] "RemoveContainer" containerID="dab691c2e17ac09767f6c75771d0548d0040d4157d938f271539703c49f2b842"
Jul 28 01:39:39.374322 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:39:39.374305    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab691c2e17ac09767f6c75771d0548d0040d4157d938f271539703c49f2b842\": container with ID starting with dab691c2e17ac09767f6c75771d0548d0040d4157d938f271539703c49f2b842 not found: ID does not exist" containerID="dab691c2e17ac09767f6c75771d0548d0040d4157d938f271539703c49f2b842"
Jul 28 01:39:39.374365 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.374324    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab691c2e17ac09767f6c75771d0548d0040d4157d938f271539703c49f2b842"} err="failed to get container status \"dab691c2e17ac09767f6c75771d0548d0040d4157d938f271539703c49f2b842\": rpc error: code = NotFound desc = could not find container \"dab691c2e17ac09767f6c75771d0548d0040d4157d938f271539703c49f2b842\": container with ID starting with dab691c2e17ac09767f6c75771d0548d0040d4157d938f271539703c49f2b842 not found: ID does not exist"
Jul 28 01:39:39.374365 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.374336    2555 scope.go:117] "RemoveContainer" containerID="d0af0a445e189ad4d835879722891ca43c8f6d43d55711a4b65a2f4e697170d0"
Jul 28 01:39:39.374600 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:39:39.374578    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0af0a445e189ad4d835879722891ca43c8f6d43d55711a4b65a2f4e697170d0\": container with ID starting with d0af0a445e189ad4d835879722891ca43c8f6d43d55711a4b65a2f4e697170d0 not found: ID does not exist" containerID="d0af0a445e189ad4d835879722891ca43c8f6d43d55711a4b65a2f4e697170d0"
Jul 28 01:39:39.374646 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.374604    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0af0a445e189ad4d835879722891ca43c8f6d43d55711a4b65a2f4e697170d0"} err="failed to get container status \"d0af0a445e189ad4d835879722891ca43c8f6d43d55711a4b65a2f4e697170d0\": rpc error: code = NotFound desc = could not find container \"d0af0a445e189ad4d835879722891ca43c8f6d43d55711a4b65a2f4e697170d0\": container with ID starting with d0af0a445e189ad4d835879722891ca43c8f6d43d55711a4b65a2f4e697170d0 not found: ID does not exist"
Jul 28 01:39:39.450939 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:39.450908    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" path="/var/lib/kubelet/pods/40393776-09b8-4e98-812f-5723c4d55a0f/volumes"
Jul 28 01:39:40.327325 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:40.327300    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:39:40.328021 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:40.327996    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 01:39:40.328269 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:40.328247    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:39:50.327955 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:50.327909    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 01:39:50.328441 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:39:50.328382    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:40:00.328691 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:00.328648    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 01:40:00.329179 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:00.329153    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:40:10.328265 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:10.328213    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 01:40:10.328716 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:10.328695    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:40:20.327899 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:20.327858    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 01:40:20.328412 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:20.328335    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:40:30.328712 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:30.328660    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 01:40:30.329152 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:30.329066    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:40:40.328725 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:40.328655    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:40:40.329026 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:40.328743    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:40:53.419870 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.419796    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-6zv2z_b7b00ab5-1811-45ca-8f56-ed6e19ec551f/kserve-container/0.log"
Jul 28 01:40:53.584710 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.584680    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"]
Jul 28 01:40:53.584950 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.584931    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kserve-container" containerID="cri-o://6cb7683fee8dd17c725f45618ea450c93807493f3e62c3361d7151b2b8a3c1d1" gracePeriod=30
Jul 28 01:40:53.585045 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.585022    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kube-rbac-proxy" containerID="cri-o://7f4b5470a5814e27ec30df1cddf0f9443c654f36893e42bbe391f2aa6062a93a" gracePeriod=30
Jul 28 01:40:53.585109 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.585021    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="agent" containerID="cri-o://e264c7ea75bc7db4e12a4eab2bb233c558af142b0633886c9804a4d6a936cd39" gracePeriod=30
Jul 28 01:40:53.647506 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.647476    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"]
Jul 28 01:40:53.647764 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.647752    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="storage-initializer"
Jul 28 01:40:53.647807 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.647767    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="storage-initializer"
Jul 28 01:40:53.647807 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.647776    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kube-rbac-proxy"
Jul 28 01:40:53.647807 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.647783    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kube-rbac-proxy"
Jul 28 01:40:53.647807 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.647797    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kserve-container"
Jul 28 01:40:53.647807 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.647803    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kserve-container"
Jul 28 01:40:53.647950 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.647811    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="agent"
Jul 28 01:40:53.647950 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.647816    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="agent"
Jul 28 01:40:53.647950 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.647863    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kserve-container"
Jul 28 01:40:53.647950 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.647873    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="agent"
Jul 28 01:40:53.647950 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.647882    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="40393776-09b8-4e98-812f-5723c4d55a0f" containerName="kube-rbac-proxy"
Jul 28 01:40:53.650966 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.650950    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:53.653235 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.653197    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 28 01:40:53.653335 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.653303    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 28 01:40:53.661018 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.660992    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"]
Jul 28 01:40:53.692904 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.692840    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"]
Jul 28 01:40:53.693082 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.693038    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z" podUID="b7b00ab5-1811-45ca-8f56-ed6e19ec551f" containerName="kserve-container" containerID="cri-o://03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46" gracePeriod=30
Jul 28 01:40:53.693167 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.693064    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z" podUID="b7b00ab5-1811-45ca-8f56-ed6e19ec551f" containerName="kube-rbac-proxy" containerID="cri-o://61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4" gracePeriod=30
Jul 28 01:40:53.785919 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.785894    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c954e24-c803-4fff-99ea-19f3511946c0-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xb9xl\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:53.786051 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.785925    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c954e24-c803-4fff-99ea-19f3511946c0-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xb9xl\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:53.786051 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.785947    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c954e24-c803-4fff-99ea-19f3511946c0-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xb9xl\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:53.786051 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.785964    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c954e24-c803-4fff-99ea-19f3511946c0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xb9xl\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:53.786051 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.786030    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtkxd\" (UniqueName: \"kubernetes.io/projected/0c954e24-c803-4fff-99ea-19f3511946c0-kube-api-access-jtkxd\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xb9xl\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:53.887359 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.887324    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtkxd\" (UniqueName: \"kubernetes.io/projected/0c954e24-c803-4fff-99ea-19f3511946c0-kube-api-access-jtkxd\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xb9xl\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:53.887486 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.887432    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c954e24-c803-4fff-99ea-19f3511946c0-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xb9xl\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:53.887486 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.887463    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c954e24-c803-4fff-99ea-19f3511946c0-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xb9xl\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:53.887627 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.887496    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c954e24-c803-4fff-99ea-19f3511946c0-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xb9xl\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:53.887627 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:40:53.887609    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 28 01:40:53.887722 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.887636    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c954e24-c803-4fff-99ea-19f3511946c0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xb9xl\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:53.887722 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:40:53.887668    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c954e24-c803-4fff-99ea-19f3511946c0-proxy-tls podName:0c954e24-c803-4fff-99ea-19f3511946c0 nodeName:}" failed. No retries permitted until 2026-07-28 01:40:54.38764671 +0000 UTC m=+901.397165138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0c954e24-c803-4fff-99ea-19f3511946c0-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-xb9xl" (UID: "0c954e24-c803-4fff-99ea-19f3511946c0") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 28 01:40:53.887910 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.887888    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c954e24-c803-4fff-99ea-19f3511946c0-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xb9xl\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:53.888262 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.888242    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c954e24-c803-4fff-99ea-19f3511946c0-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xb9xl\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:53.888262 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.888251    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c954e24-c803-4fff-99ea-19f3511946c0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xb9xl\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:53.898221 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.898195    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtkxd\" (UniqueName: \"kubernetes.io/projected/0c954e24-c803-4fff-99ea-19f3511946c0-kube-api-access-jtkxd\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xb9xl\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:53.931624 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:53.931598    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:40:54.089771 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.089742    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-kube-rbac-proxy-sar-config\") pod \"b7b00ab5-1811-45ca-8f56-ed6e19ec551f\" (UID: \"b7b00ab5-1811-45ca-8f56-ed6e19ec551f\") "
Jul 28 01:40:54.089935 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.089806    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcq9z\" (UniqueName: \"kubernetes.io/projected/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-kube-api-access-kcq9z\") pod \"b7b00ab5-1811-45ca-8f56-ed6e19ec551f\" (UID: \"b7b00ab5-1811-45ca-8f56-ed6e19ec551f\") "
Jul 28 01:40:54.089935 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.089840    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-proxy-tls\") pod \"b7b00ab5-1811-45ca-8f56-ed6e19ec551f\" (UID: \"b7b00ab5-1811-45ca-8f56-ed6e19ec551f\") "
Jul 28 01:40:54.090082 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.090060    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b7b00ab5-1811-45ca-8f56-ed6e19ec551f" (UID: "b7b00ab5-1811-45ca-8f56-ed6e19ec551f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:40:54.091924 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.091903    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-kube-api-access-kcq9z" (OuterVolumeSpecName: "kube-api-access-kcq9z") pod "b7b00ab5-1811-45ca-8f56-ed6e19ec551f" (UID: "b7b00ab5-1811-45ca-8f56-ed6e19ec551f"). InnerVolumeSpecName "kube-api-access-kcq9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:40:54.092012 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.091925    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b7b00ab5-1811-45ca-8f56-ed6e19ec551f" (UID: "b7b00ab5-1811-45ca-8f56-ed6e19ec551f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:40:54.190395 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.190373    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:40:54.190395 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.190395    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcq9z\" (UniqueName: \"kubernetes.io/projected/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-kube-api-access-kcq9z\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:40:54.190544 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.190407    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7b00ab5-1811-45ca-8f56-ed6e19ec551f-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:40:54.392175 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.392115    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c954e24-c803-4fff-99ea-19f3511946c0-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xb9xl\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:54.394362 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.394346    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c954e24-c803-4fff-99ea-19f3511946c0-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xb9xl\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:54.561512 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.561479    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:40:54.598828 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.598798    2555 generic.go:356] "Generic (PLEG): container finished" podID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerID="7f4b5470a5814e27ec30df1cddf0f9443c654f36893e42bbe391f2aa6062a93a" exitCode=2
Jul 28 01:40:54.598958 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.598870    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" event={"ID":"05b64515-c5d5-42fe-97bb-5b16b6a48e4e","Type":"ContainerDied","Data":"7f4b5470a5814e27ec30df1cddf0f9443c654f36893e42bbe391f2aa6062a93a"}
Jul 28 01:40:54.600718 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.600687    2555 generic.go:356] "Generic (PLEG): container finished" podID="b7b00ab5-1811-45ca-8f56-ed6e19ec551f" containerID="61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4" exitCode=2
Jul 28 01:40:54.600718 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.600711    2555 generic.go:356] "Generic (PLEG): container finished" podID="b7b00ab5-1811-45ca-8f56-ed6e19ec551f" containerID="03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46" exitCode=2
Jul 28 01:40:54.600843 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.600731    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"
Jul 28 01:40:54.600843 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.600785    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z" event={"ID":"b7b00ab5-1811-45ca-8f56-ed6e19ec551f","Type":"ContainerDied","Data":"61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4"}
Jul 28 01:40:54.600905 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.600845    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z" event={"ID":"b7b00ab5-1811-45ca-8f56-ed6e19ec551f","Type":"ContainerDied","Data":"03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46"}
Jul 28 01:40:54.600905 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.600877    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z" event={"ID":"b7b00ab5-1811-45ca-8f56-ed6e19ec551f","Type":"ContainerDied","Data":"64779e999d0edc48d8b53cf22c7defafd9bb600e495e0d5db143d35113ac0ba6"}
Jul 28 01:40:54.600965 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.600911    2555 scope.go:117] "RemoveContainer" containerID="61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4"
Jul 28 01:40:54.610396 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.610381    2555 scope.go:117] "RemoveContainer" containerID="03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46"
Jul 28 01:40:54.620668 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.620123    2555 scope.go:117] "RemoveContainer" containerID="61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4"
Jul 28 01:40:54.620668 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:40:54.620446    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4\": container with ID starting with 61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4 not found: ID does not exist" containerID="61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4"
Jul 28 01:40:54.620668 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.620478    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4"} err="failed to get container status \"61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4\": rpc error: code = NotFound desc = could not find container \"61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4\": container with ID starting with 61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4 not found: ID does not exist"
Jul 28 01:40:54.620668 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.620502    2555 scope.go:117] "RemoveContainer" containerID="03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46"
Jul 28 01:40:54.620981 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:40:54.620892    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46\": container with ID starting with 03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46 not found: ID does not exist" containerID="03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46"
Jul 28 01:40:54.620981 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.620934    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46"} err="failed to get container status \"03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46\": rpc error: code = NotFound desc = could not find container \"03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46\": container with ID starting with 03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46 not found: ID does not exist"
Jul 28 01:40:54.620981 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.620957    2555 scope.go:117] "RemoveContainer" containerID="61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4"
Jul 28 01:40:54.621217 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.621197    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4"} err="failed to get container status \"61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4\": rpc error: code = NotFound desc = could not find container \"61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4\": container with ID starting with 61520c98f64836f69c905770ee1fdeeaadff4fdbe84f1bf62837fdf9054a2ac4 not found: ID does not exist"
Jul 28 01:40:54.621304 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.621219    2555 scope.go:117] "RemoveContainer" containerID="03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46"
Jul 28 01:40:54.621971 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.621468    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46"} err="failed to get container status \"03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46\": rpc error: code = NotFound desc = could not find container \"03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46\": container with ID starting with 03e257ae0ec3f4bb3200411407b696a372503622c1517169cbc12e363a3bab46 not found: ID does not exist"
Jul 28 01:40:54.622495 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.622125    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"]
Jul 28 01:40:54.626780 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.626760    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zv2z"]
Jul 28 01:40:54.694893 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:54.694769    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"]
Jul 28 01:40:54.697170 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:40:54.697147    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c954e24_c803_4fff_99ea_19f3511946c0.slice/crio-521c0e911d3316e1a1009c7225c30b0aaeafd709aa52042df87d7bfdb34e154f WatchSource:0}: Error finding container 521c0e911d3316e1a1009c7225c30b0aaeafd709aa52042df87d7bfdb34e154f: Status 404 returned error can't find the container with id 521c0e911d3316e1a1009c7225c30b0aaeafd709aa52042df87d7bfdb34e154f
Jul 28 01:40:55.323812 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:55.323766    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Jul 28 01:40:55.451643 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:55.451607    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7b00ab5-1811-45ca-8f56-ed6e19ec551f" path="/var/lib/kubelet/pods/b7b00ab5-1811-45ca-8f56-ed6e19ec551f/volumes"
Jul 28 01:40:55.606027 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:55.605942    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl" event={"ID":"0c954e24-c803-4fff-99ea-19f3511946c0","Type":"ContainerStarted","Data":"e2277babf1bfb33c6a7468b4c67130e302747f3a9f9f580ae129bed7625ee837"}
Jul 28 01:40:55.606027 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:55.606010    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl" event={"ID":"0c954e24-c803-4fff-99ea-19f3511946c0","Type":"ContainerStarted","Data":"521c0e911d3316e1a1009c7225c30b0aaeafd709aa52042df87d7bfdb34e154f"}
Jul 28 01:40:57.615841 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:57.615815    2555 generic.go:356] "Generic (PLEG): container finished" podID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerID="6cb7683fee8dd17c725f45618ea450c93807493f3e62c3361d7151b2b8a3c1d1" exitCode=0
Jul 28 01:40:57.616102 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:40:57.615851    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" event={"ID":"05b64515-c5d5-42fe-97bb-5b16b6a48e4e","Type":"ContainerDied","Data":"6cb7683fee8dd17c725f45618ea450c93807493f3e62c3361d7151b2b8a3c1d1"}
Jul 28 01:41:00.323635 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:00.323599    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Jul 28 01:41:00.327960 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:00.327920    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 01:41:00.328229 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:00.328208    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:41:05.324125 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:05.324083    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Jul 28 01:41:05.324486 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:05.324208    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:41:10.323357 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:10.323313    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Jul 28 01:41:10.328715 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:10.328684    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 01:41:10.328977 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:10.328955    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:41:15.324187 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:15.324142    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Jul 28 01:41:20.324157 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:20.324112    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Jul 28 01:41:20.328601 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:20.328552    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 01:41:20.328724 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:20.328707    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:41:20.328946 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:20.328917    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:41:20.329028 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:20.329018    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:41:23.702128 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.702090    2555 generic.go:356] "Generic (PLEG): container finished" podID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerID="e264c7ea75bc7db4e12a4eab2bb233c558af142b0633886c9804a4d6a936cd39" exitCode=0
Jul 28 01:41:23.702548 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.702143    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" event={"ID":"05b64515-c5d5-42fe-97bb-5b16b6a48e4e","Type":"ContainerDied","Data":"e264c7ea75bc7db4e12a4eab2bb233c558af142b0633886c9804a4d6a936cd39"}
Jul 28 01:41:23.735818 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.735795    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:41:23.788657 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.788630    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-proxy-tls\") pod \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") "
Jul 28 01:41:23.788823 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.788805    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfrxb\" (UniqueName: \"kubernetes.io/projected/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kube-api-access-qfrxb\") pod \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") "
Jul 28 01:41:23.788878 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.788855    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kube-rbac-proxy-sar-config\") pod \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") "
Jul 28 01:41:23.788930 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.788887    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kserve-provision-location\") pod \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") "
Jul 28 01:41:23.789012 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.788990    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-cabundle-cert\") pod \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\" (UID: \"05b64515-c5d5-42fe-97bb-5b16b6a48e4e\") "
Jul 28 01:41:23.789246 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.789166    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "05b64515-c5d5-42fe-97bb-5b16b6a48e4e" (UID: "05b64515-c5d5-42fe-97bb-5b16b6a48e4e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:23.789348 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.789207    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "05b64515-c5d5-42fe-97bb-5b16b6a48e4e" (UID: "05b64515-c5d5-42fe-97bb-5b16b6a48e4e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:41:23.789348 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.789299    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "05b64515-c5d5-42fe-97bb-5b16b6a48e4e" (UID: "05b64515-c5d5-42fe-97bb-5b16b6a48e4e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:41:23.790910 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.790877    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kube-api-access-qfrxb" (OuterVolumeSpecName: "kube-api-access-qfrxb") pod "05b64515-c5d5-42fe-97bb-5b16b6a48e4e" (UID: "05b64515-c5d5-42fe-97bb-5b16b6a48e4e"). InnerVolumeSpecName "kube-api-access-qfrxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:23.790995 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.790927    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "05b64515-c5d5-42fe-97bb-5b16b6a48e4e" (UID: "05b64515-c5d5-42fe-97bb-5b16b6a48e4e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:41:23.889461 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.889409    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:41:23.889461 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.889431    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfrxb\" (UniqueName: \"kubernetes.io/projected/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kube-api-access-qfrxb\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:41:23.889461 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.889441    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:41:23.889461 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.889449    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:41:23.889461 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:23.889457    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05b64515-c5d5-42fe-97bb-5b16b6a48e4e-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:41:24.707540 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:24.707495    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t" event={"ID":"05b64515-c5d5-42fe-97bb-5b16b6a48e4e","Type":"ContainerDied","Data":"4a0930406707d3e5b2f77de5f7af846f99fe9a0c2604c73b5daee99cc5adecb0"}
Jul 28 01:41:24.707993 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:24.707584    2555 scope.go:117] "RemoveContainer" containerID="e264c7ea75bc7db4e12a4eab2bb233c558af142b0633886c9804a4d6a936cd39"
Jul 28 01:41:24.707993 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:24.707503    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"
Jul 28 01:41:24.716035 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:24.716020    2555 scope.go:117] "RemoveContainer" containerID="7f4b5470a5814e27ec30df1cddf0f9443c654f36893e42bbe391f2aa6062a93a"
Jul 28 01:41:24.722776 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:24.722759    2555 scope.go:117] "RemoveContainer" containerID="6cb7683fee8dd17c725f45618ea450c93807493f3e62c3361d7151b2b8a3c1d1"
Jul 28 01:41:24.729025 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:24.729003    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"]
Jul 28 01:41:24.731573 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:24.731503    2555 scope.go:117] "RemoveContainer" containerID="1b5e6d7597609d1b4680f8253aee4a3ef022487268075d93afcd392d08db9322"
Jul 28 01:41:24.733782 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:24.733765    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-h8b5t"]
Jul 28 01:41:25.451789 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:25.451750    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" path="/var/lib/kubelet/pods/05b64515-c5d5-42fe-97bb-5b16b6a48e4e/volumes"
Jul 28 01:41:25.713872 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:25.713807    2555 generic.go:356] "Generic (PLEG): container finished" podID="0c954e24-c803-4fff-99ea-19f3511946c0" containerID="e2277babf1bfb33c6a7468b4c67130e302747f3a9f9f580ae129bed7625ee837" exitCode=0
Jul 28 01:41:25.714181 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:41:25.713874    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl" event={"ID":"0c954e24-c803-4fff-99ea-19f3511946c0","Type":"ContainerDied","Data":"e2277babf1bfb33c6a7468b4c67130e302747f3a9f9f580ae129bed7625ee837"}
Jul 28 01:43:32.195376 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:43:32.195344    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl" event={"ID":"0c954e24-c803-4fff-99ea-19f3511946c0","Type":"ContainerStarted","Data":"233c77e77383c2531b3bab91dc6dcb9859cea3d6fa3a96995aec5573c806ff18"}
Jul 28 01:43:32.195800 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:43:32.195408    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl" event={"ID":"0c954e24-c803-4fff-99ea-19f3511946c0","Type":"ContainerStarted","Data":"53c41ca879d86ec70e64a36c7c5af5138d9eea09a9d4fd8e6646a2cf6c457361"}
Jul 28 01:43:32.195800 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:43:32.195511    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:43:32.224993 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:43:32.224946    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl" podStartSLOduration=2.419070098 podStartE2EDuration="2m39.224931874s" totalImagesPullingTime="2m5.995860154s" totalInitContainerRuntime="30.810001622s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:43:32.223475494 +0000 UTC m=+1059.232994121" watchObservedRunningTime="2026-07-28 01:43:32.224931874 +0000 UTC m=+1059.234450297"
Jul 28 01:43:33.199837 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:43:33.199802    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:43:33.200917 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:43:33.200888    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl" podUID="0c954e24-c803-4fff-99ea-19f3511946c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 28 01:43:34.202999 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:43:34.202958    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl" podUID="0c954e24-c803-4fff-99ea-19f3511946c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 28 01:43:39.207884 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:43:39.207855    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:43:39.208563 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:43:39.208483    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl" podUID="0c954e24-c803-4fff-99ea-19f3511946c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 28 01:43:49.208755 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:43:49.208673    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:43:59.749671 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:43:59.749640    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"]
Jul 28 01:43:59.750166 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:43:59.749944    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl" podUID="0c954e24-c803-4fff-99ea-19f3511946c0" containerName="kserve-container" containerID="cri-o://53c41ca879d86ec70e64a36c7c5af5138d9eea09a9d4fd8e6646a2cf6c457361" gracePeriod=30
Jul 28 01:43:59.750166 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:43:59.749989    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl" podUID="0c954e24-c803-4fff-99ea-19f3511946c0" containerName="kube-rbac-proxy" containerID="cri-o://233c77e77383c2531b3bab91dc6dcb9859cea3d6fa3a96995aec5573c806ff18" gracePeriod=30
Jul 28 01:44:00.289302 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:00.289268    2555 generic.go:356] "Generic (PLEG): container finished" podID="0c954e24-c803-4fff-99ea-19f3511946c0" containerID="233c77e77383c2531b3bab91dc6dcb9859cea3d6fa3a96995aec5573c806ff18" exitCode=2
Jul 28 01:44:00.289464 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:00.289343    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl" event={"ID":"0c954e24-c803-4fff-99ea-19f3511946c0","Type":"ContainerDied","Data":"233c77e77383c2531b3bab91dc6dcb9859cea3d6fa3a96995aec5573c806ff18"}
Jul 28 01:44:01.117598 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.117545    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:44:01.157607 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.157584    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtkxd\" (UniqueName: \"kubernetes.io/projected/0c954e24-c803-4fff-99ea-19f3511946c0-kube-api-access-jtkxd\") pod \"0c954e24-c803-4fff-99ea-19f3511946c0\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") "
Jul 28 01:44:01.157759 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.157622    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c954e24-c803-4fff-99ea-19f3511946c0-proxy-tls\") pod \"0c954e24-c803-4fff-99ea-19f3511946c0\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") "
Jul 28 01:44:01.157759 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.157642    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c954e24-c803-4fff-99ea-19f3511946c0-kube-rbac-proxy-sar-config\") pod \"0c954e24-c803-4fff-99ea-19f3511946c0\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") "
Jul 28 01:44:01.157759 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.157664    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c954e24-c803-4fff-99ea-19f3511946c0-cabundle-cert\") pod \"0c954e24-c803-4fff-99ea-19f3511946c0\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") "
Jul 28 01:44:01.157924 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.157786    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c954e24-c803-4fff-99ea-19f3511946c0-kserve-provision-location\") pod \"0c954e24-c803-4fff-99ea-19f3511946c0\" (UID: \"0c954e24-c803-4fff-99ea-19f3511946c0\") "
Jul 28 01:44:01.158020 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.157998    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c954e24-c803-4fff-99ea-19f3511946c0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0c954e24-c803-4fff-99ea-19f3511946c0" (UID: "0c954e24-c803-4fff-99ea-19f3511946c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:44:01.158068 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.158004    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c954e24-c803-4fff-99ea-19f3511946c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0c954e24-c803-4fff-99ea-19f3511946c0" (UID: "0c954e24-c803-4fff-99ea-19f3511946c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:44:01.159700 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.159671    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c954e24-c803-4fff-99ea-19f3511946c0-kube-api-access-jtkxd" (OuterVolumeSpecName: "kube-api-access-jtkxd") pod "0c954e24-c803-4fff-99ea-19f3511946c0" (UID: "0c954e24-c803-4fff-99ea-19f3511946c0"). InnerVolumeSpecName "kube-api-access-jtkxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:44:01.160069 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.160051    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c954e24-c803-4fff-99ea-19f3511946c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0c954e24-c803-4fff-99ea-19f3511946c0" (UID: "0c954e24-c803-4fff-99ea-19f3511946c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:44:01.163037 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.163008    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c954e24-c803-4fff-99ea-19f3511946c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0c954e24-c803-4fff-99ea-19f3511946c0" (UID: "0c954e24-c803-4fff-99ea-19f3511946c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:44:01.259071 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.259044    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c954e24-c803-4fff-99ea-19f3511946c0-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:44:01.259071 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.259065    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c954e24-c803-4fff-99ea-19f3511946c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:44:01.259071 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.259075    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c954e24-c803-4fff-99ea-19f3511946c0-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:44:01.259226 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.259083    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c954e24-c803-4fff-99ea-19f3511946c0-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:44:01.259226 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.259092    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtkxd\" (UniqueName: \"kubernetes.io/projected/0c954e24-c803-4fff-99ea-19f3511946c0-kube-api-access-jtkxd\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:44:01.294409 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.294378    2555 generic.go:356] "Generic (PLEG): container finished" podID="0c954e24-c803-4fff-99ea-19f3511946c0" containerID="53c41ca879d86ec70e64a36c7c5af5138d9eea09a9d4fd8e6646a2cf6c457361" exitCode=0
Jul 28 01:44:01.294504 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.294474    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl" event={"ID":"0c954e24-c803-4fff-99ea-19f3511946c0","Type":"ContainerDied","Data":"53c41ca879d86ec70e64a36c7c5af5138d9eea09a9d4fd8e6646a2cf6c457361"}
Jul 28 01:44:01.294504 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.294497    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"
Jul 28 01:44:01.294612 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.294589    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl" event={"ID":"0c954e24-c803-4fff-99ea-19f3511946c0","Type":"ContainerDied","Data":"521c0e911d3316e1a1009c7225c30b0aaeafd709aa52042df87d7bfdb34e154f"}
Jul 28 01:44:01.294648 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.294639    2555 scope.go:117] "RemoveContainer" containerID="233c77e77383c2531b3bab91dc6dcb9859cea3d6fa3a96995aec5573c806ff18"
Jul 28 01:44:01.302737 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.302723    2555 scope.go:117] "RemoveContainer" containerID="53c41ca879d86ec70e64a36c7c5af5138d9eea09a9d4fd8e6646a2cf6c457361"
Jul 28 01:44:01.309503 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.309485    2555 scope.go:117] "RemoveContainer" containerID="e2277babf1bfb33c6a7468b4c67130e302747f3a9f9f580ae129bed7625ee837"
Jul 28 01:44:01.314429 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.314411    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"]
Jul 28 01:44:01.317127 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.317106    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xb9xl"]
Jul 28 01:44:01.317481 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.317463    2555 scope.go:117] "RemoveContainer" containerID="233c77e77383c2531b3bab91dc6dcb9859cea3d6fa3a96995aec5573c806ff18"
Jul 28 01:44:01.317735 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:44:01.317718    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"233c77e77383c2531b3bab91dc6dcb9859cea3d6fa3a96995aec5573c806ff18\": container with ID starting with 233c77e77383c2531b3bab91dc6dcb9859cea3d6fa3a96995aec5573c806ff18 not found: ID does not exist" containerID="233c77e77383c2531b3bab91dc6dcb9859cea3d6fa3a96995aec5573c806ff18"
Jul 28 01:44:01.317789 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.317743    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233c77e77383c2531b3bab91dc6dcb9859cea3d6fa3a96995aec5573c806ff18"} err="failed to get container status \"233c77e77383c2531b3bab91dc6dcb9859cea3d6fa3a96995aec5573c806ff18\": rpc error: code = NotFound desc = could not find container \"233c77e77383c2531b3bab91dc6dcb9859cea3d6fa3a96995aec5573c806ff18\": container with ID starting with 233c77e77383c2531b3bab91dc6dcb9859cea3d6fa3a96995aec5573c806ff18 not found: ID does not exist"
Jul 28 01:44:01.317789 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.317762    2555 scope.go:117] "RemoveContainer" containerID="53c41ca879d86ec70e64a36c7c5af5138d9eea09a9d4fd8e6646a2cf6c457361"
Jul 28 01:44:01.317999 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:44:01.317979    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c41ca879d86ec70e64a36c7c5af5138d9eea09a9d4fd8e6646a2cf6c457361\": container with ID starting with 53c41ca879d86ec70e64a36c7c5af5138d9eea09a9d4fd8e6646a2cf6c457361 not found: ID does not exist" containerID="53c41ca879d86ec70e64a36c7c5af5138d9eea09a9d4fd8e6646a2cf6c457361"
Jul 28 01:44:01.318042 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.318005    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c41ca879d86ec70e64a36c7c5af5138d9eea09a9d4fd8e6646a2cf6c457361"} err="failed to get container status \"53c41ca879d86ec70e64a36c7c5af5138d9eea09a9d4fd8e6646a2cf6c457361\": rpc error: code = NotFound desc = could not find container \"53c41ca879d86ec70e64a36c7c5af5138d9eea09a9d4fd8e6646a2cf6c457361\": container with ID starting with 53c41ca879d86ec70e64a36c7c5af5138d9eea09a9d4fd8e6646a2cf6c457361 not found: ID does not exist"
Jul 28 01:44:01.318042 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.318019    2555 scope.go:117] "RemoveContainer" containerID="e2277babf1bfb33c6a7468b4c67130e302747f3a9f9f580ae129bed7625ee837"
Jul 28 01:44:01.318196 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:44:01.318181    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2277babf1bfb33c6a7468b4c67130e302747f3a9f9f580ae129bed7625ee837\": container with ID starting with e2277babf1bfb33c6a7468b4c67130e302747f3a9f9f580ae129bed7625ee837 not found: ID does not exist" containerID="e2277babf1bfb33c6a7468b4c67130e302747f3a9f9f580ae129bed7625ee837"
Jul 28 01:44:01.318234 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.318199    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2277babf1bfb33c6a7468b4c67130e302747f3a9f9f580ae129bed7625ee837"} err="failed to get container status \"e2277babf1bfb33c6a7468b4c67130e302747f3a9f9f580ae129bed7625ee837\": rpc error: code = NotFound desc = could not find container \"e2277babf1bfb33c6a7468b4c67130e302747f3a9f9f580ae129bed7625ee837\": container with ID starting with e2277babf1bfb33c6a7468b4c67130e302747f3a9f9f580ae129bed7625ee837 not found: ID does not exist"
Jul 28 01:44:01.450946 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:01.450890    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c954e24-c803-4fff-99ea-19f3511946c0" path="/var/lib/kubelet/pods/0c954e24-c803-4fff-99ea-19f3511946c0/volumes"
Jul 28 01:44:04.843801 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.843774    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"]
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.843998    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kserve-container"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844008    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kserve-container"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844019    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7b00ab5-1811-45ca-8f56-ed6e19ec551f" containerName="kserve-container"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844024    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7b00ab5-1811-45ca-8f56-ed6e19ec551f" containerName="kserve-container"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844030    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c954e24-c803-4fff-99ea-19f3511946c0" containerName="storage-initializer"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844036    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c954e24-c803-4fff-99ea-19f3511946c0" containerName="storage-initializer"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844049    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="agent"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844054    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="agent"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844060    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kube-rbac-proxy"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844065    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kube-rbac-proxy"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844074    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="storage-initializer"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844079    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="storage-initializer"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844086    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c954e24-c803-4fff-99ea-19f3511946c0" containerName="kube-rbac-proxy"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844090    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c954e24-c803-4fff-99ea-19f3511946c0" containerName="kube-rbac-proxy"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844100    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7b00ab5-1811-45ca-8f56-ed6e19ec551f" containerName="kube-rbac-proxy"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844105    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7b00ab5-1811-45ca-8f56-ed6e19ec551f" containerName="kube-rbac-proxy"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844111    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c954e24-c803-4fff-99ea-19f3511946c0" containerName="kserve-container"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844117    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c954e24-c803-4fff-99ea-19f3511946c0" containerName="kserve-container"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844151    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7b00ab5-1811-45ca-8f56-ed6e19ec551f" containerName="kube-rbac-proxy"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844158    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kube-rbac-proxy"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844165    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="kserve-container"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844170    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="05b64515-c5d5-42fe-97bb-5b16b6a48e4e" containerName="agent"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844175    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7b00ab5-1811-45ca-8f56-ed6e19ec551f" containerName="kserve-container"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844181    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c954e24-c803-4fff-99ea-19f3511946c0" containerName="kube-rbac-proxy"
Jul 28 01:44:04.844236 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.844187    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c954e24-c803-4fff-99ea-19f3511946c0" containerName="kserve-container"
Jul 28 01:44:04.872826 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.872800    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"]
Jul 28 01:44:04.872948 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.872899    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:04.875622 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.875590    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 28 01:44:04.875622 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.875605    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 28 01:44:04.875801 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.875651    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 01:44:04.875801 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.875677    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 01:44:04.875801 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.875606    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 01:44:04.875929 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.875896    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 01:44:04.876586 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.876569    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 01:44:04.982190 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.982169    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zn48s\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:04.982308 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.982200    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwsxm\" (UniqueName: \"kubernetes.io/projected/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kube-api-access-vwsxm\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zn48s\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:04.982308 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.982224    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zn48s\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:04.982422 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.982303    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zn48s\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:04.982422 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:04.982378    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zn48s\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:05.083639 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:05.083619    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zn48s\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:05.083750 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:05.083650    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwsxm\" (UniqueName: \"kubernetes.io/projected/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kube-api-access-vwsxm\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zn48s\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:05.083750 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:05.083670    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zn48s\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:05.083750 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:05.083689    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zn48s\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:05.083750 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:05.083725    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zn48s\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:05.083939 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:44:05.083810    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 28 01:44:05.083939 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:44:05.083906    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-proxy-tls podName:5bb1b68e-355b-431d-a76f-6aae5f1bc0f2 nodeName:}" failed. No retries permitted until 2026-07-28 01:44:05.583885882 +0000 UTC m=+1092.593404289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-proxy-tls") pod "isvc-autogluon-v2-predictor-9684b4cd9-zn48s" (UID: "5bb1b68e-355b-431d-a76f-6aae5f1bc0f2") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 28 01:44:05.084122 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:05.084107    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zn48s\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:05.084263 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:05.084246    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zn48s\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:05.084333 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:05.084318    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zn48s\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:05.092384 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:05.092357    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwsxm\" (UniqueName: \"kubernetes.io/projected/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kube-api-access-vwsxm\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zn48s\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:05.587404 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:05.587376    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zn48s\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:05.589763 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:05.589744    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zn48s\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:05.782768 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:05.782739    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:05.926818 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:05.926787    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"]
Jul 28 01:44:05.929842 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:44:05.929811    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb1b68e_355b_431d_a76f_6aae5f1bc0f2.slice/crio-9d6b1f55f6fced28a283a30158ca296f47ebc99531e78635477db29ca9a18d3f WatchSource:0}: Error finding container 9d6b1f55f6fced28a283a30158ca296f47ebc99531e78635477db29ca9a18d3f: Status 404 returned error can't find the container with id 9d6b1f55f6fced28a283a30158ca296f47ebc99531e78635477db29ca9a18d3f
Jul 28 01:44:06.314286 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:06.314254    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s" event={"ID":"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2","Type":"ContainerStarted","Data":"2136bcaf3a5464d97dcc52f0849be4208105642dae853b479bcde0fdd5743280"}
Jul 28 01:44:06.314452 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:06.314315    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s" event={"ID":"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2","Type":"ContainerStarted","Data":"9d6b1f55f6fced28a283a30158ca296f47ebc99531e78635477db29ca9a18d3f"}
Jul 28 01:44:26.379492 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:26.379460    2555 generic.go:356] "Generic (PLEG): container finished" podID="5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" containerID="2136bcaf3a5464d97dcc52f0849be4208105642dae853b479bcde0fdd5743280" exitCode=0
Jul 28 01:44:26.379901 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:26.379556    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s" event={"ID":"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2","Type":"ContainerDied","Data":"2136bcaf3a5464d97dcc52f0849be4208105642dae853b479bcde0fdd5743280"}
Jul 28 01:44:27.386244 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:27.386211    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s" event={"ID":"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2","Type":"ContainerStarted","Data":"4805182823fd02e2024eb7e34493d7c46c586ec7ec4c811688ddbde954ebdd89"}
Jul 28 01:44:27.386692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:27.386269    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s" event={"ID":"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2","Type":"ContainerStarted","Data":"3ca9cbee84b96d308202a9897398c0374f30409563df61a849559763b6c76e83"}
Jul 28 01:44:27.386692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:27.386463    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:28.389286 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:28.389254    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:34.399340 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:34.399313    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:44:34.429567 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:44:34.429498    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s" podStartSLOduration=10.184757661 podStartE2EDuration="30.429482922s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.244725261s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:44:27.406541862 +0000 UTC m=+1114.416060293" watchObservedRunningTime="2026-07-28 01:44:34.429482922 +0000 UTC m=+1121.439001368"
Jul 28 01:45:58.450973 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:45:58.450939    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:46:09.489706 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:09.489667    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s" podUID="5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.21:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 01:46:10.820054 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:10.820024    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"]
Jul 28 01:46:10.820443 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:10.820269    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s" podUID="5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" containerName="kserve-container" containerID="cri-o://3ca9cbee84b96d308202a9897398c0374f30409563df61a849559763b6c76e83" gracePeriod=30
Jul 28 01:46:10.820443 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:10.820297    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s" podUID="5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" containerName="kube-rbac-proxy" containerID="cri-o://4805182823fd02e2024eb7e34493d7c46c586ec7ec4c811688ddbde954ebdd89" gracePeriod=30
Jul 28 01:46:11.730566 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:11.730512    2555 generic.go:356] "Generic (PLEG): container finished" podID="5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" containerID="4805182823fd02e2024eb7e34493d7c46c586ec7ec4c811688ddbde954ebdd89" exitCode=2
Jul 28 01:46:11.730566 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:11.730554    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s" event={"ID":"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2","Type":"ContainerDied","Data":"4805182823fd02e2024eb7e34493d7c46c586ec7ec4c811688ddbde954ebdd89"}
Jul 28 01:46:12.054607 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.054576    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:46:12.145084 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.145060    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-cabundle-cert\") pod \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") "
Jul 28 01:46:12.145220 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.145090    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-proxy-tls\") pod \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") "
Jul 28 01:46:12.145220 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.145116    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kube-rbac-proxy-sar-config\") pod \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") "
Jul 28 01:46:12.145220 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.145176    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwsxm\" (UniqueName: \"kubernetes.io/projected/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kube-api-access-vwsxm\") pod \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") "
Jul 28 01:46:12.145220 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.145203    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kserve-provision-location\") pod \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\" (UID: \"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2\") "
Jul 28 01:46:12.145485 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.145447    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" (UID: "5bb1b68e-355b-431d-a76f-6aae5f1bc0f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:46:12.145795 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.145769    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" (UID: "5bb1b68e-355b-431d-a76f-6aae5f1bc0f2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:46:12.147187 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.147167    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" (UID: "5bb1b68e-355b-431d-a76f-6aae5f1bc0f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:46:12.147383 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.147358    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kube-api-access-vwsxm" (OuterVolumeSpecName: "kube-api-access-vwsxm") pod "5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" (UID: "5bb1b68e-355b-431d-a76f-6aae5f1bc0f2"). InnerVolumeSpecName "kube-api-access-vwsxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:46:12.149926 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.149904    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" (UID: "5bb1b68e-355b-431d-a76f-6aae5f1bc0f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:46:12.245806 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.245777    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwsxm\" (UniqueName: \"kubernetes.io/projected/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kube-api-access-vwsxm\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:46:12.245806 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.245805    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:46:12.245932 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.245819    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:46:12.245932 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.245831    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:46:12.245932 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.245845    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:46:12.735235 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.735156    2555 generic.go:356] "Generic (PLEG): container finished" podID="5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" containerID="3ca9cbee84b96d308202a9897398c0374f30409563df61a849559763b6c76e83" exitCode=0
Jul 28 01:46:12.735235 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.735212    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"
Jul 28 01:46:12.735424 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.735212    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s" event={"ID":"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2","Type":"ContainerDied","Data":"3ca9cbee84b96d308202a9897398c0374f30409563df61a849559763b6c76e83"}
Jul 28 01:46:12.735424 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.735339    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s" event={"ID":"5bb1b68e-355b-431d-a76f-6aae5f1bc0f2","Type":"ContainerDied","Data":"9d6b1f55f6fced28a283a30158ca296f47ebc99531e78635477db29ca9a18d3f"}
Jul 28 01:46:12.735424 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.735369    2555 scope.go:117] "RemoveContainer" containerID="4805182823fd02e2024eb7e34493d7c46c586ec7ec4c811688ddbde954ebdd89"
Jul 28 01:46:12.743338 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.743320    2555 scope.go:117] "RemoveContainer" containerID="3ca9cbee84b96d308202a9897398c0374f30409563df61a849559763b6c76e83"
Jul 28 01:46:12.752264 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.752248    2555 scope.go:117] "RemoveContainer" containerID="2136bcaf3a5464d97dcc52f0849be4208105642dae853b479bcde0fdd5743280"
Jul 28 01:46:12.755551 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.755508    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"]
Jul 28 01:46:12.758851 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.758830    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zn48s"]
Jul 28 01:46:12.759841 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.759825    2555 scope.go:117] "RemoveContainer" containerID="4805182823fd02e2024eb7e34493d7c46c586ec7ec4c811688ddbde954ebdd89"
Jul 28 01:46:12.760125 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:46:12.760107    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4805182823fd02e2024eb7e34493d7c46c586ec7ec4c811688ddbde954ebdd89\": container with ID starting with 4805182823fd02e2024eb7e34493d7c46c586ec7ec4c811688ddbde954ebdd89 not found: ID does not exist" containerID="4805182823fd02e2024eb7e34493d7c46c586ec7ec4c811688ddbde954ebdd89"
Jul 28 01:46:12.760172 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.760133    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4805182823fd02e2024eb7e34493d7c46c586ec7ec4c811688ddbde954ebdd89"} err="failed to get container status \"4805182823fd02e2024eb7e34493d7c46c586ec7ec4c811688ddbde954ebdd89\": rpc error: code = NotFound desc = could not find container \"4805182823fd02e2024eb7e34493d7c46c586ec7ec4c811688ddbde954ebdd89\": container with ID starting with 4805182823fd02e2024eb7e34493d7c46c586ec7ec4c811688ddbde954ebdd89 not found: ID does not exist"
Jul 28 01:46:12.760172 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.760150    2555 scope.go:117] "RemoveContainer" containerID="3ca9cbee84b96d308202a9897398c0374f30409563df61a849559763b6c76e83"
Jul 28 01:46:12.760388 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:46:12.760371    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca9cbee84b96d308202a9897398c0374f30409563df61a849559763b6c76e83\": container with ID starting with 3ca9cbee84b96d308202a9897398c0374f30409563df61a849559763b6c76e83 not found: ID does not exist" containerID="3ca9cbee84b96d308202a9897398c0374f30409563df61a849559763b6c76e83"
Jul 28 01:46:12.760431 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.760394    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca9cbee84b96d308202a9897398c0374f30409563df61a849559763b6c76e83"} err="failed to get container status \"3ca9cbee84b96d308202a9897398c0374f30409563df61a849559763b6c76e83\": rpc error: code = NotFound desc = could not find container \"3ca9cbee84b96d308202a9897398c0374f30409563df61a849559763b6c76e83\": container with ID starting with 3ca9cbee84b96d308202a9897398c0374f30409563df61a849559763b6c76e83 not found: ID does not exist"
Jul 28 01:46:12.760431 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.760409    2555 scope.go:117] "RemoveContainer" containerID="2136bcaf3a5464d97dcc52f0849be4208105642dae853b479bcde0fdd5743280"
Jul 28 01:46:12.760629 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:46:12.760613    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2136bcaf3a5464d97dcc52f0849be4208105642dae853b479bcde0fdd5743280\": container with ID starting with 2136bcaf3a5464d97dcc52f0849be4208105642dae853b479bcde0fdd5743280 not found: ID does not exist" containerID="2136bcaf3a5464d97dcc52f0849be4208105642dae853b479bcde0fdd5743280"
Jul 28 01:46:12.760687 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:12.760636    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2136bcaf3a5464d97dcc52f0849be4208105642dae853b479bcde0fdd5743280"} err="failed to get container status \"2136bcaf3a5464d97dcc52f0849be4208105642dae853b479bcde0fdd5743280\": rpc error: code = NotFound desc = could not find container \"2136bcaf3a5464d97dcc52f0849be4208105642dae853b479bcde0fdd5743280\": container with ID starting with 2136bcaf3a5464d97dcc52f0849be4208105642dae853b479bcde0fdd5743280 not found: ID does not exist"
Jul 28 01:46:13.450822 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:13.450783    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" path="/var/lib/kubelet/pods/5bb1b68e-355b-431d-a76f-6aae5f1bc0f2/volumes"
Jul 28 01:46:15.968674 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.968644    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"]
Jul 28 01:46:15.969015 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.968901    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" containerName="kube-rbac-proxy"
Jul 28 01:46:15.969015 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.968911    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" containerName="kube-rbac-proxy"
Jul 28 01:46:15.969015 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.968920    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" containerName="kserve-container"
Jul 28 01:46:15.969015 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.968926    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" containerName="kserve-container"
Jul 28 01:46:15.969015 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.968935    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" containerName="storage-initializer"
Jul 28 01:46:15.969015 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.968940    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" containerName="storage-initializer"
Jul 28 01:46:15.969015 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.968993    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" containerName="kube-rbac-proxy"
Jul 28 01:46:15.969015 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.969000    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb1b68e-355b-431d-a76f-6aae5f1bc0f2" containerName="kserve-container"
Jul 28 01:46:15.972380 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.972362    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:15.974808 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.974782    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 01:46:15.974808 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.974799    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 01:46:15.975064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.975049    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 28 01:46:15.975153 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.975057    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 01:46:15.975153 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.975075    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 28 01:46:15.975993 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.975974    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 01:46:15.976083 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.976013    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 01:46:15.982935 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:15.982915    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"]
Jul 28 01:46:16.066723 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.066694    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1a96136-55b7-448a-8039-61ec06baae87-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:16.066827 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.066725    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1a96136-55b7-448a-8039-61ec06baae87-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:16.066827 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.066751    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1a96136-55b7-448a-8039-61ec06baae87-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:16.066827 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.066817    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1a96136-55b7-448a-8039-61ec06baae87-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:16.066971 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.066856    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j2fp\" (UniqueName: \"kubernetes.io/projected/c1a96136-55b7-448a-8039-61ec06baae87-kube-api-access-2j2fp\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:16.167171 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.167147    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1a96136-55b7-448a-8039-61ec06baae87-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:16.167290 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.167186    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1a96136-55b7-448a-8039-61ec06baae87-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:16.167290 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.167207    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j2fp\" (UniqueName: \"kubernetes.io/projected/c1a96136-55b7-448a-8039-61ec06baae87-kube-api-access-2j2fp\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:16.167290 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.167233    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1a96136-55b7-448a-8039-61ec06baae87-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:16.167290 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.167254    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1a96136-55b7-448a-8039-61ec06baae87-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:16.167636 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.167615    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1a96136-55b7-448a-8039-61ec06baae87-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:16.167942 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.167924    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1a96136-55b7-448a-8039-61ec06baae87-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:16.167987 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.167926    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1a96136-55b7-448a-8039-61ec06baae87-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:16.169689 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.169674    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1a96136-55b7-448a-8039-61ec06baae87-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:16.175857 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.175832    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j2fp\" (UniqueName: \"kubernetes.io/projected/c1a96136-55b7-448a-8039-61ec06baae87-kube-api-access-2j2fp\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:16.281890 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.281867    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:16.406608 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.406574    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"]
Jul 28 01:46:16.408808 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:46:16.408780    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a96136_55b7_448a_8039_61ec06baae87.slice/crio-a89f1bdbc22679d9a240696ac7e1dda5ac63bf3616968bcd03d95a301d8db817 WatchSource:0}: Error finding container a89f1bdbc22679d9a240696ac7e1dda5ac63bf3616968bcd03d95a301d8db817: Status 404 returned error can't find the container with id a89f1bdbc22679d9a240696ac7e1dda5ac63bf3616968bcd03d95a301d8db817
Jul 28 01:46:16.755771 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.755736    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p" event={"ID":"c1a96136-55b7-448a-8039-61ec06baae87","Type":"ContainerStarted","Data":"13b24c70327a413281b45d535ebdd83fab2bc900237ddb702055e5d4812f9a8e"}
Jul 28 01:46:16.755923 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:16.755811    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p" event={"ID":"c1a96136-55b7-448a-8039-61ec06baae87","Type":"ContainerStarted","Data":"a89f1bdbc22679d9a240696ac7e1dda5ac63bf3616968bcd03d95a301d8db817"}
Jul 28 01:46:22.777099 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:22.777064    2555 generic.go:356] "Generic (PLEG): container finished" podID="c1a96136-55b7-448a-8039-61ec06baae87" containerID="13b24c70327a413281b45d535ebdd83fab2bc900237ddb702055e5d4812f9a8e" exitCode=0
Jul 28 01:46:22.777648 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:22.777128    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p" event={"ID":"c1a96136-55b7-448a-8039-61ec06baae87","Type":"ContainerDied","Data":"13b24c70327a413281b45d535ebdd83fab2bc900237ddb702055e5d4812f9a8e"}
Jul 28 01:46:23.783351 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:23.783314    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p" event={"ID":"c1a96136-55b7-448a-8039-61ec06baae87","Type":"ContainerStarted","Data":"54eca612652d5ae1e4bec9d4322971da68ed6cc6b53a3f7763a667180fade9d5"}
Jul 28 01:46:23.783722 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:23.783376    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p" event={"ID":"c1a96136-55b7-448a-8039-61ec06baae87","Type":"ContainerStarted","Data":"5ecdf7fed2164884ec32a10afa1f4d11a7d04ab319b8623ab0ee160423c049bc"}
Jul 28 01:46:23.783722 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:23.783573    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:23.783722 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:23.783620    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:46:23.801298 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:23.801255    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p" podStartSLOduration=2.925935734 podStartE2EDuration="8.801243294s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.87530756s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:46:23.800222348 +0000 UTC m=+1230.809740826" watchObservedRunningTime="2026-07-28 01:46:23.801243294 +0000 UTC m=+1230.810761718"
Jul 28 01:46:29.793225 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:46:29.793197    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:47:54.451129 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:47:54.451087    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:48:02.806865 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:02.806833    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"]
Jul 28 01:48:02.807362 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:02.807067    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p" podUID="c1a96136-55b7-448a-8039-61ec06baae87" containerName="kserve-container" containerID="cri-o://5ecdf7fed2164884ec32a10afa1f4d11a7d04ab319b8623ab0ee160423c049bc" gracePeriod=30
Jul 28 01:48:02.807362 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:02.807160    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p" podUID="c1a96136-55b7-448a-8039-61ec06baae87" containerName="kube-rbac-proxy" containerID="cri-o://54eca612652d5ae1e4bec9d4322971da68ed6cc6b53a3f7763a667180fade9d5" gracePeriod=30
Jul 28 01:48:03.126868 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:03.126786    2555 generic.go:356] "Generic (PLEG): container finished" podID="c1a96136-55b7-448a-8039-61ec06baae87" containerID="54eca612652d5ae1e4bec9d4322971da68ed6cc6b53a3f7763a667180fade9d5" exitCode=2
Jul 28 01:48:03.127004 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:03.126867    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p" event={"ID":"c1a96136-55b7-448a-8039-61ec06baae87","Type":"ContainerDied","Data":"54eca612652d5ae1e4bec9d4322971da68ed6cc6b53a3f7763a667180fade9d5"}
Jul 28 01:48:04.132815 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.132785    2555 generic.go:356] "Generic (PLEG): container finished" podID="c1a96136-55b7-448a-8039-61ec06baae87" containerID="5ecdf7fed2164884ec32a10afa1f4d11a7d04ab319b8623ab0ee160423c049bc" exitCode=0
Jul 28 01:48:04.133107 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.132856    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p" event={"ID":"c1a96136-55b7-448a-8039-61ec06baae87","Type":"ContainerDied","Data":"5ecdf7fed2164884ec32a10afa1f4d11a7d04ab319b8623ab0ee160423c049bc"}
Jul 28 01:48:04.146584 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.146562    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:48:04.251407 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.251365    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j2fp\" (UniqueName: \"kubernetes.io/projected/c1a96136-55b7-448a-8039-61ec06baae87-kube-api-access-2j2fp\") pod \"c1a96136-55b7-448a-8039-61ec06baae87\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") "
Jul 28 01:48:04.251613 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.251424    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1a96136-55b7-448a-8039-61ec06baae87-kserve-provision-location\") pod \"c1a96136-55b7-448a-8039-61ec06baae87\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") "
Jul 28 01:48:04.251613 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.251448    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1a96136-55b7-448a-8039-61ec06baae87-proxy-tls\") pod \"c1a96136-55b7-448a-8039-61ec06baae87\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") "
Jul 28 01:48:04.251613 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.251473    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1a96136-55b7-448a-8039-61ec06baae87-cabundle-cert\") pod \"c1a96136-55b7-448a-8039-61ec06baae87\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") "
Jul 28 01:48:04.251613 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.251560    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1a96136-55b7-448a-8039-61ec06baae87-kube-rbac-proxy-sar-config\") pod \"c1a96136-55b7-448a-8039-61ec06baae87\" (UID: \"c1a96136-55b7-448a-8039-61ec06baae87\") "
Jul 28 01:48:04.251957 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.251902    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a96136-55b7-448a-8039-61ec06baae87-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c1a96136-55b7-448a-8039-61ec06baae87" (UID: "c1a96136-55b7-448a-8039-61ec06baae87"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:48:04.251957 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.251945    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a96136-55b7-448a-8039-61ec06baae87-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c1a96136-55b7-448a-8039-61ec06baae87" (UID: "c1a96136-55b7-448a-8039-61ec06baae87"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:48:04.253705 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.253674    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a96136-55b7-448a-8039-61ec06baae87-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c1a96136-55b7-448a-8039-61ec06baae87" (UID: "c1a96136-55b7-448a-8039-61ec06baae87"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:48:04.253801 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.253749    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a96136-55b7-448a-8039-61ec06baae87-kube-api-access-2j2fp" (OuterVolumeSpecName: "kube-api-access-2j2fp") pod "c1a96136-55b7-448a-8039-61ec06baae87" (UID: "c1a96136-55b7-448a-8039-61ec06baae87"). InnerVolumeSpecName "kube-api-access-2j2fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:48:04.257743 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.257716    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a96136-55b7-448a-8039-61ec06baae87-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1a96136-55b7-448a-8039-61ec06baae87" (UID: "c1a96136-55b7-448a-8039-61ec06baae87"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:48:04.352452 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.352414    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1a96136-55b7-448a-8039-61ec06baae87-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:48:04.352452 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.352450    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1a96136-55b7-448a-8039-61ec06baae87-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:48:04.352452 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.352460    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j2fp\" (UniqueName: \"kubernetes.io/projected/c1a96136-55b7-448a-8039-61ec06baae87-kube-api-access-2j2fp\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:48:04.352666 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.352471    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1a96136-55b7-448a-8039-61ec06baae87-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:48:04.352666 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:04.352480    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1a96136-55b7-448a-8039-61ec06baae87-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:48:05.137996 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:05.137962    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p" event={"ID":"c1a96136-55b7-448a-8039-61ec06baae87","Type":"ContainerDied","Data":"a89f1bdbc22679d9a240696ac7e1dda5ac63bf3616968bcd03d95a301d8db817"}
Jul 28 01:48:05.138413 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:05.138011    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"
Jul 28 01:48:05.138413 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:05.138031    2555 scope.go:117] "RemoveContainer" containerID="54eca612652d5ae1e4bec9d4322971da68ed6cc6b53a3f7763a667180fade9d5"
Jul 28 01:48:05.146449 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:05.146433    2555 scope.go:117] "RemoveContainer" containerID="5ecdf7fed2164884ec32a10afa1f4d11a7d04ab319b8623ab0ee160423c049bc"
Jul 28 01:48:05.153485 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:05.153469    2555 scope.go:117] "RemoveContainer" containerID="13b24c70327a413281b45d535ebdd83fab2bc900237ddb702055e5d4812f9a8e"
Jul 28 01:48:05.160355 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:05.160338    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"]
Jul 28 01:48:05.163926 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:05.163906    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tzd5p"]
Jul 28 01:48:05.451490 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:05.451412    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a96136-55b7-448a-8039-61ec06baae87" path="/var/lib/kubelet/pods/c1a96136-55b7-448a-8039-61ec06baae87/volumes"
Jul 28 01:48:07.923357 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.923323    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"]
Jul 28 01:48:07.923755 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.923588    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a96136-55b7-448a-8039-61ec06baae87" containerName="kserve-container"
Jul 28 01:48:07.923755 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.923601    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a96136-55b7-448a-8039-61ec06baae87" containerName="kserve-container"
Jul 28 01:48:07.923755 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.923610    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a96136-55b7-448a-8039-61ec06baae87" containerName="kube-rbac-proxy"
Jul 28 01:48:07.923755 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.923616    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a96136-55b7-448a-8039-61ec06baae87" containerName="kube-rbac-proxy"
Jul 28 01:48:07.923755 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.923627    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a96136-55b7-448a-8039-61ec06baae87" containerName="storage-initializer"
Jul 28 01:48:07.923755 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.923633    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a96136-55b7-448a-8039-61ec06baae87" containerName="storage-initializer"
Jul 28 01:48:07.923755 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.923697    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a96136-55b7-448a-8039-61ec06baae87" containerName="kserve-container"
Jul 28 01:48:07.923755 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.923705    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a96136-55b7-448a-8039-61ec06baae87" containerName="kube-rbac-proxy"
Jul 28 01:48:07.928468 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.928443    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:07.931382 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.931358    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 01:48:07.931540 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.931393    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 01:48:07.931540 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.931400    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 01:48:07.931690 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.931663    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 28 01:48:07.931767 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.931753    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 01:48:07.931832 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.931799    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 28 01:48:07.932424 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.932409    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 01:48:07.937386 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:07.937362    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"]
Jul 28 01:48:08.077396 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.077358    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:08.077624 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.077408    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:08.077624 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.077435    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsx8b\" (UniqueName: \"kubernetes.io/projected/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kube-api-access-xsx8b\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:08.077624 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.077489    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:08.077624 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.077559    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:08.178280 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.178200    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:08.178280 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.178247    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:08.178280 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.178272    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:08.178562 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.178387    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsx8b\" (UniqueName: \"kubernetes.io/projected/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kube-api-access-xsx8b\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:08.178562 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.178447    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:08.178726 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.178692    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:08.178979 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.178954    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:08.179057 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.179006    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:08.180809 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.180792    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:08.188820 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.188794    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsx8b\" (UniqueName: \"kubernetes.io/projected/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kube-api-access-xsx8b\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:08.238805 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.238764    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:08.372453 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:08.372429    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"]
Jul 28 01:48:08.374617 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:48:08.374578    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb851c9fa_84cc_4e8a_bbc6_e4ee17b07a10.slice/crio-467b77f272d8777650b68b91ef44184de401af74db0946b6dd89c134a6fa521d WatchSource:0}: Error finding container 467b77f272d8777650b68b91ef44184de401af74db0946b6dd89c134a6fa521d: Status 404 returned error can't find the container with id 467b77f272d8777650b68b91ef44184de401af74db0946b6dd89c134a6fa521d
Jul 28 01:48:09.153305 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:09.153272    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s" event={"ID":"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10","Type":"ContainerStarted","Data":"f10478fafb1547ac99343924defa5a89ab686af5c7b11ad07c3d2ea969079487"}
Jul 28 01:48:09.153679 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:09.153335    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s" event={"ID":"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10","Type":"ContainerStarted","Data":"467b77f272d8777650b68b91ef44184de401af74db0946b6dd89c134a6fa521d"}
Jul 28 01:48:29.221485 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:29.221402    2555 generic.go:356] "Generic (PLEG): container finished" podID="b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" containerID="f10478fafb1547ac99343924defa5a89ab686af5c7b11ad07c3d2ea969079487" exitCode=0
Jul 28 01:48:29.221485 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:29.221476    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s" event={"ID":"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10","Type":"ContainerDied","Data":"f10478fafb1547ac99343924defa5a89ab686af5c7b11ad07c3d2ea969079487"}
Jul 28 01:48:30.228566 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:30.228516    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s" event={"ID":"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10","Type":"ContainerStarted","Data":"c99b9d57a6af04156867968a57f5407fba9bceebc141efabefd0688a8d96ca52"}
Jul 28 01:48:30.228970 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:30.228593    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s" event={"ID":"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10","Type":"ContainerStarted","Data":"ec59bb0b9586552b47abcbc0e85a4dee3ef2c950c9be959d4fb9e5f5cbfc6371"}
Jul 28 01:48:30.228970 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:30.228815    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:30.228970 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:30.228953    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:48:30.248031 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:30.247972    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s" podStartSLOduration=2.910635513 podStartE2EDuration="23.247955179s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.337319666s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:48:30.246773207 +0000 UTC m=+1357.256291657" watchObservedRunningTime="2026-07-28 01:48:30.247955179 +0000 UTC m=+1357.257473606"
Jul 28 01:48:36.238188 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:48:36.238160    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:50:02.451137 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:02.451067    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:50:13.488707 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:13.488665    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s" podUID="b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.23:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 01:50:14.233499 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:14.233467    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"]
Jul 28 01:50:14.233991 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:14.233764    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s" podUID="b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" containerName="kserve-container" containerID="cri-o://ec59bb0b9586552b47abcbc0e85a4dee3ef2c950c9be959d4fb9e5f5cbfc6371" gracePeriod=30
Jul 28 01:50:14.233991 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:14.233820    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s" podUID="b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" containerName="kube-rbac-proxy" containerID="cri-o://c99b9d57a6af04156867968a57f5407fba9bceebc141efabefd0688a8d96ca52" gracePeriod=30
Jul 28 01:50:14.586092 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:14.586056    2555 generic.go:356] "Generic (PLEG): container finished" podID="b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" containerID="c99b9d57a6af04156867968a57f5407fba9bceebc141efabefd0688a8d96ca52" exitCode=2
Jul 28 01:50:14.586617 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:14.586100    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s" event={"ID":"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10","Type":"ContainerDied","Data":"c99b9d57a6af04156867968a57f5407fba9bceebc141efabefd0688a8d96ca52"}
Jul 28 01:50:15.572407 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.572382    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:50:15.591180 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.590978    2555 generic.go:356] "Generic (PLEG): container finished" podID="b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" containerID="ec59bb0b9586552b47abcbc0e85a4dee3ef2c950c9be959d4fb9e5f5cbfc6371" exitCode=0
Jul 28 01:50:15.591180 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.591060    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s" event={"ID":"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10","Type":"ContainerDied","Data":"ec59bb0b9586552b47abcbc0e85a4dee3ef2c950c9be959d4fb9e5f5cbfc6371"}
Jul 28 01:50:15.591180 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.591118    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s" event={"ID":"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10","Type":"ContainerDied","Data":"467b77f272d8777650b68b91ef44184de401af74db0946b6dd89c134a6fa521d"}
Jul 28 01:50:15.591180 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.591158    2555 scope.go:117] "RemoveContainer" containerID="c99b9d57a6af04156867968a57f5407fba9bceebc141efabefd0688a8d96ca52"
Jul 28 01:50:15.591745 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.591266    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"
Jul 28 01:50:15.600810 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.600793    2555 scope.go:117] "RemoveContainer" containerID="ec59bb0b9586552b47abcbc0e85a4dee3ef2c950c9be959d4fb9e5f5cbfc6371"
Jul 28 01:50:15.608999 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.608960    2555 scope.go:117] "RemoveContainer" containerID="f10478fafb1547ac99343924defa5a89ab686af5c7b11ad07c3d2ea969079487"
Jul 28 01:50:15.615794 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.615770    2555 scope.go:117] "RemoveContainer" containerID="c99b9d57a6af04156867968a57f5407fba9bceebc141efabefd0688a8d96ca52"
Jul 28 01:50:15.616037 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:50:15.616019    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99b9d57a6af04156867968a57f5407fba9bceebc141efabefd0688a8d96ca52\": container with ID starting with c99b9d57a6af04156867968a57f5407fba9bceebc141efabefd0688a8d96ca52 not found: ID does not exist" containerID="c99b9d57a6af04156867968a57f5407fba9bceebc141efabefd0688a8d96ca52"
Jul 28 01:50:15.616114 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.616050    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99b9d57a6af04156867968a57f5407fba9bceebc141efabefd0688a8d96ca52"} err="failed to get container status \"c99b9d57a6af04156867968a57f5407fba9bceebc141efabefd0688a8d96ca52\": rpc error: code = NotFound desc = could not find container \"c99b9d57a6af04156867968a57f5407fba9bceebc141efabefd0688a8d96ca52\": container with ID starting with c99b9d57a6af04156867968a57f5407fba9bceebc141efabefd0688a8d96ca52 not found: ID does not exist"
Jul 28 01:50:15.616114 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.616075    2555 scope.go:117] "RemoveContainer" containerID="ec59bb0b9586552b47abcbc0e85a4dee3ef2c950c9be959d4fb9e5f5cbfc6371"
Jul 28 01:50:15.616338 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:50:15.616319    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec59bb0b9586552b47abcbc0e85a4dee3ef2c950c9be959d4fb9e5f5cbfc6371\": container with ID starting with ec59bb0b9586552b47abcbc0e85a4dee3ef2c950c9be959d4fb9e5f5cbfc6371 not found: ID does not exist" containerID="ec59bb0b9586552b47abcbc0e85a4dee3ef2c950c9be959d4fb9e5f5cbfc6371"
Jul 28 01:50:15.616383 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.616348    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec59bb0b9586552b47abcbc0e85a4dee3ef2c950c9be959d4fb9e5f5cbfc6371"} err="failed to get container status \"ec59bb0b9586552b47abcbc0e85a4dee3ef2c950c9be959d4fb9e5f5cbfc6371\": rpc error: code = NotFound desc = could not find container \"ec59bb0b9586552b47abcbc0e85a4dee3ef2c950c9be959d4fb9e5f5cbfc6371\": container with ID starting with ec59bb0b9586552b47abcbc0e85a4dee3ef2c950c9be959d4fb9e5f5cbfc6371 not found: ID does not exist"
Jul 28 01:50:15.616383 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.616365    2555 scope.go:117] "RemoveContainer" containerID="f10478fafb1547ac99343924defa5a89ab686af5c7b11ad07c3d2ea969079487"
Jul 28 01:50:15.616614 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:50:15.616593    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10478fafb1547ac99343924defa5a89ab686af5c7b11ad07c3d2ea969079487\": container with ID starting with f10478fafb1547ac99343924defa5a89ab686af5c7b11ad07c3d2ea969079487 not found: ID does not exist" containerID="f10478fafb1547ac99343924defa5a89ab686af5c7b11ad07c3d2ea969079487"
Jul 28 01:50:15.616671 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.616620    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10478fafb1547ac99343924defa5a89ab686af5c7b11ad07c3d2ea969079487"} err="failed to get container status \"f10478fafb1547ac99343924defa5a89ab686af5c7b11ad07c3d2ea969079487\": rpc error: code = NotFound desc = could not find container \"f10478fafb1547ac99343924defa5a89ab686af5c7b11ad07c3d2ea969079487\": container with ID starting with f10478fafb1547ac99343924defa5a89ab686af5c7b11ad07c3d2ea969079487 not found: ID does not exist"
Jul 28 01:50:15.684407 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.684387    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsx8b\" (UniqueName: \"kubernetes.io/projected/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kube-api-access-xsx8b\") pod \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") "
Jul 28 01:50:15.684504 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.684447    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-proxy-tls\") pod \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") "
Jul 28 01:50:15.684504 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.684471    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-cabundle-cert\") pod \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") "
Jul 28 01:50:15.684632 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.684611    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kserve-provision-location\") pod \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") "
Jul 28 01:50:15.684692 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.684648    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kube-rbac-proxy-sar-config\") pod \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\" (UID: \"b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10\") "
Jul 28 01:50:15.684881 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.684782    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" (UID: "b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:50:15.685038 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.685012    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" (UID: "b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:50:15.686432 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.686411    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kube-api-access-xsx8b" (OuterVolumeSpecName: "kube-api-access-xsx8b") pod "b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" (UID: "b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10"). InnerVolumeSpecName "kube-api-access-xsx8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:50:15.686606 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.686578    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" (UID: "b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:50:15.688631 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.688612    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" (UID: "b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:50:15.785610 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.785584    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsx8b\" (UniqueName: \"kubernetes.io/projected/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kube-api-access-xsx8b\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:50:15.785610 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.785605    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:50:15.785610 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.785614    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:50:15.785771 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.785622    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:50:15.785771 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.785631    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:50:15.912849 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.912817    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"]
Jul 28 01:50:15.917346 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:15.917322    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rpt5s"]
Jul 28 01:50:17.451937 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:17.451902    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" path="/var/lib/kubelet/pods/b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10/volumes"
Jul 28 01:50:19.384085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.384047    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"]
Jul 28 01:50:19.384458 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.384298    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" containerName="kserve-container"
Jul 28 01:50:19.384458 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.384309    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" containerName="kserve-container"
Jul 28 01:50:19.384458 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.384321    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" containerName="storage-initializer"
Jul 28 01:50:19.384458 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.384326    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" containerName="storage-initializer"
Jul 28 01:50:19.384458 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.384335    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" containerName="kube-rbac-proxy"
Jul 28 01:50:19.384458 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.384340    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" containerName="kube-rbac-proxy"
Jul 28 01:50:19.384458 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.384379    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" containerName="kube-rbac-proxy"
Jul 28 01:50:19.384458 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.384388    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b851c9fa-84cc-4e8a-bbc6-e4ee17b07a10" containerName="kserve-container"
Jul 28 01:50:19.388831 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.388812    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:19.391188 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.391165    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 01:50:19.391188 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.391193    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 01:50:19.391413 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.391192    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 28 01:50:19.391413 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.391419    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 01:50:19.391545 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.391508    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 01:50:19.392253 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.392240    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 28 01:50:19.392351 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.392333    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 01:50:19.396820 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.396798    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"]
Jul 28 01:50:19.513782 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.513745    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:19.513978 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.513793    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:19.513978 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.513824    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:19.513978 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.513846    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:19.513978 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.513869    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27rw6\" (UniqueName: \"kubernetes.io/projected/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kube-api-access-27rw6\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:19.614356 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.614320    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:19.614356 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.614359    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:19.614630 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.614379    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:19.614630 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.614411    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rw6\" (UniqueName: \"kubernetes.io/projected/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kube-api-access-27rw6\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:19.614630 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.614462    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:19.614630 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:50:19.614583    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 28 01:50:19.614874 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:50:19.614681    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-proxy-tls podName:838f2fa9-b4cc-40db-8d71-c9b8f2d75769 nodeName:}" failed. No retries permitted until 2026-07-28 01:50:20.114660568 +0000 UTC m=+1467.124178971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj" (UID: "838f2fa9-b4cc-40db-8d71-c9b8f2d75769") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 28 01:50:19.614947 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.614880    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:19.615091 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.615070    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:19.615165 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.615121    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:19.624540 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:19.624501    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27rw6\" (UniqueName: \"kubernetes.io/projected/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kube-api-access-27rw6\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:20.118173 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:20.118129    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:20.120668 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:20.120639    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:20.300643 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:20.300591    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:20.435672 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:20.435648    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"]
Jul 28 01:50:20.438201 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:50:20.438173    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838f2fa9_b4cc_40db_8d71_c9b8f2d75769.slice/crio-abbdc51b55a35aa572e712998fb9254117a42a248dfccb0159709042566a8ec6 WatchSource:0}: Error finding container abbdc51b55a35aa572e712998fb9254117a42a248dfccb0159709042566a8ec6: Status 404 returned error can't find the container with id abbdc51b55a35aa572e712998fb9254117a42a248dfccb0159709042566a8ec6
Jul 28 01:50:20.610974 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:20.610931    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj" event={"ID":"838f2fa9-b4cc-40db-8d71-c9b8f2d75769","Type":"ContainerStarted","Data":"ac76e8c198e050c01db771c6243a45a03a683acaa5f8d867e4344f37b5dae87e"}
Jul 28 01:50:20.611156 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:20.610985    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj" event={"ID":"838f2fa9-b4cc-40db-8d71-c9b8f2d75769","Type":"ContainerStarted","Data":"abbdc51b55a35aa572e712998fb9254117a42a248dfccb0159709042566a8ec6"}
Jul 28 01:50:30.646645 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:30.646609    2555 generic.go:356] "Generic (PLEG): container finished" podID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" containerID="ac76e8c198e050c01db771c6243a45a03a683acaa5f8d867e4344f37b5dae87e" exitCode=0
Jul 28 01:50:30.647108 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:30.646690    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj" event={"ID":"838f2fa9-b4cc-40db-8d71-c9b8f2d75769","Type":"ContainerDied","Data":"ac76e8c198e050c01db771c6243a45a03a683acaa5f8d867e4344f37b5dae87e"}
Jul 28 01:50:31.653408 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:31.653363    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj" event={"ID":"838f2fa9-b4cc-40db-8d71-c9b8f2d75769","Type":"ContainerStarted","Data":"83dccccadb867583c324add99963982c46c91a1d63b92e85e54aafaa9db43a7e"}
Jul 28 01:50:31.653780 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:31.653451    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj" event={"ID":"838f2fa9-b4cc-40db-8d71-c9b8f2d75769","Type":"ContainerStarted","Data":"c5323945bdf3e06a0b61c946e796455d50905da8c2ebbb2d987aa650f61e6458"}
Jul 28 01:50:31.653780 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:31.653688    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:31.672718 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:31.672665    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj" podStartSLOduration=2.968284477 podStartE2EDuration="12.672645513s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.704361036s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:50:31.671651362 +0000 UTC m=+1478.681169813" watchObservedRunningTime="2026-07-28 01:50:31.672645513 +0000 UTC m=+1478.682163938"
Jul 28 01:50:32.657896 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:32.657866    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:32.659208 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:32.659181    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj" podUID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 01:50:33.668351 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:33.668307    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj" podUID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 01:50:38.674075 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:38.674043    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:50:38.674748 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:38.674723    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj" podUID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 01:50:48.675321 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:50:48.675290    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:51:01.538155 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:01.538110    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"]
Jul 28 01:51:01.538610 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:01.538415    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj" podUID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" containerName="kserve-container" containerID="cri-o://c5323945bdf3e06a0b61c946e796455d50905da8c2ebbb2d987aa650f61e6458" gracePeriod=30
Jul 28 01:51:01.538610 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:01.538438    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj" podUID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" containerName="kube-rbac-proxy" containerID="cri-o://83dccccadb867583c324add99963982c46c91a1d63b92e85e54aafaa9db43a7e" gracePeriod=30
Jul 28 01:51:01.767172 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:01.767140    2555 generic.go:356] "Generic (PLEG): container finished" podID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" containerID="83dccccadb867583c324add99963982c46c91a1d63b92e85e54aafaa9db43a7e" exitCode=2
Jul 28 01:51:01.767343 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:01.767177    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj" event={"ID":"838f2fa9-b4cc-40db-8d71-c9b8f2d75769","Type":"ContainerDied","Data":"83dccccadb867583c324add99963982c46c91a1d63b92e85e54aafaa9db43a7e"}
Jul 28 01:51:03.189020 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.188997    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:51:03.241537 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.241494    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-proxy-tls\") pod \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") "
Jul 28 01:51:03.241693 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.241551    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kube-rbac-proxy-sar-config\") pod \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") "
Jul 28 01:51:03.241693 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.241604    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kserve-provision-location\") pod \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") "
Jul 28 01:51:03.241693 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.241630    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-cabundle-cert\") pod \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") "
Jul 28 01:51:03.241693 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.241670    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27rw6\" (UniqueName: \"kubernetes.io/projected/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kube-api-access-27rw6\") pod \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\" (UID: \"838f2fa9-b4cc-40db-8d71-c9b8f2d75769\") "
Jul 28 01:51:03.241982 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.241959    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "838f2fa9-b4cc-40db-8d71-c9b8f2d75769" (UID: "838f2fa9-b4cc-40db-8d71-c9b8f2d75769"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:51:03.242024 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.242000    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "838f2fa9-b4cc-40db-8d71-c9b8f2d75769" (UID: "838f2fa9-b4cc-40db-8d71-c9b8f2d75769"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:51:03.242637 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.242613    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "838f2fa9-b4cc-40db-8d71-c9b8f2d75769" (UID: "838f2fa9-b4cc-40db-8d71-c9b8f2d75769"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:51:03.243739 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.243708    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "838f2fa9-b4cc-40db-8d71-c9b8f2d75769" (UID: "838f2fa9-b4cc-40db-8d71-c9b8f2d75769"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:51:03.243863 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.243844    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kube-api-access-27rw6" (OuterVolumeSpecName: "kube-api-access-27rw6") pod "838f2fa9-b4cc-40db-8d71-c9b8f2d75769" (UID: "838f2fa9-b4cc-40db-8d71-c9b8f2d75769"). InnerVolumeSpecName "kube-api-access-27rw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:51:03.342456 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.342416    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:51:03.342456 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.342453    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:51:03.342456 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.342464    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27rw6\" (UniqueName: \"kubernetes.io/projected/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kube-api-access-27rw6\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:51:03.342456 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.342473    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:51:03.342718 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.342481    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/838f2fa9-b4cc-40db-8d71-c9b8f2d75769-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:51:03.776242 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.776210    2555 generic.go:356] "Generic (PLEG): container finished" podID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" containerID="c5323945bdf3e06a0b61c946e796455d50905da8c2ebbb2d987aa650f61e6458" exitCode=0
Jul 28 01:51:03.776391 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.776249    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj" event={"ID":"838f2fa9-b4cc-40db-8d71-c9b8f2d75769","Type":"ContainerDied","Data":"c5323945bdf3e06a0b61c946e796455d50905da8c2ebbb2d987aa650f61e6458"}
Jul 28 01:51:03.776391 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.776279    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"
Jul 28 01:51:03.776391 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.776319    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj" event={"ID":"838f2fa9-b4cc-40db-8d71-c9b8f2d75769","Type":"ContainerDied","Data":"abbdc51b55a35aa572e712998fb9254117a42a248dfccb0159709042566a8ec6"}
Jul 28 01:51:03.776391 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.776347    2555 scope.go:117] "RemoveContainer" containerID="83dccccadb867583c324add99963982c46c91a1d63b92e85e54aafaa9db43a7e"
Jul 28 01:51:03.784154 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.784136    2555 scope.go:117] "RemoveContainer" containerID="c5323945bdf3e06a0b61c946e796455d50905da8c2ebbb2d987aa650f61e6458"
Jul 28 01:51:03.790988 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.790973    2555 scope.go:117] "RemoveContainer" containerID="ac76e8c198e050c01db771c6243a45a03a683acaa5f8d867e4344f37b5dae87e"
Jul 28 01:51:03.794228 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.794205    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"]
Jul 28 01:51:03.797682 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.797660    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fqltj"]
Jul 28 01:51:03.798736 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.798715    2555 scope.go:117] "RemoveContainer" containerID="83dccccadb867583c324add99963982c46c91a1d63b92e85e54aafaa9db43a7e"
Jul 28 01:51:03.799006 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:51:03.798987    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83dccccadb867583c324add99963982c46c91a1d63b92e85e54aafaa9db43a7e\": container with ID starting with 83dccccadb867583c324add99963982c46c91a1d63b92e85e54aafaa9db43a7e not found: ID does not exist" containerID="83dccccadb867583c324add99963982c46c91a1d63b92e85e54aafaa9db43a7e"
Jul 28 01:51:03.799054 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.799015    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83dccccadb867583c324add99963982c46c91a1d63b92e85e54aafaa9db43a7e"} err="failed to get container status \"83dccccadb867583c324add99963982c46c91a1d63b92e85e54aafaa9db43a7e\": rpc error: code = NotFound desc = could not find container \"83dccccadb867583c324add99963982c46c91a1d63b92e85e54aafaa9db43a7e\": container with ID starting with 83dccccadb867583c324add99963982c46c91a1d63b92e85e54aafaa9db43a7e not found: ID does not exist"
Jul 28 01:51:03.799054 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.799032    2555 scope.go:117] "RemoveContainer" containerID="c5323945bdf3e06a0b61c946e796455d50905da8c2ebbb2d987aa650f61e6458"
Jul 28 01:51:03.799293 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:51:03.799272    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5323945bdf3e06a0b61c946e796455d50905da8c2ebbb2d987aa650f61e6458\": container with ID starting with c5323945bdf3e06a0b61c946e796455d50905da8c2ebbb2d987aa650f61e6458 not found: ID does not exist" containerID="c5323945bdf3e06a0b61c946e796455d50905da8c2ebbb2d987aa650f61e6458"
Jul 28 01:51:03.799359 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.799299    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5323945bdf3e06a0b61c946e796455d50905da8c2ebbb2d987aa650f61e6458"} err="failed to get container status \"c5323945bdf3e06a0b61c946e796455d50905da8c2ebbb2d987aa650f61e6458\": rpc error: code = NotFound desc = could not find container \"c5323945bdf3e06a0b61c946e796455d50905da8c2ebbb2d987aa650f61e6458\": container with ID starting with c5323945bdf3e06a0b61c946e796455d50905da8c2ebbb2d987aa650f61e6458 not found: ID does not exist"
Jul 28 01:51:03.799359 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.799314    2555 scope.go:117] "RemoveContainer" containerID="ac76e8c198e050c01db771c6243a45a03a683acaa5f8d867e4344f37b5dae87e"
Jul 28 01:51:03.799574 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:51:03.799557    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac76e8c198e050c01db771c6243a45a03a683acaa5f8d867e4344f37b5dae87e\": container with ID starting with ac76e8c198e050c01db771c6243a45a03a683acaa5f8d867e4344f37b5dae87e not found: ID does not exist" containerID="ac76e8c198e050c01db771c6243a45a03a683acaa5f8d867e4344f37b5dae87e"
Jul 28 01:51:03.799635 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:03.799579    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac76e8c198e050c01db771c6243a45a03a683acaa5f8d867e4344f37b5dae87e"} err="failed to get container status \"ac76e8c198e050c01db771c6243a45a03a683acaa5f8d867e4344f37b5dae87e\": rpc error: code = NotFound desc = could not find container \"ac76e8c198e050c01db771c6243a45a03a683acaa5f8d867e4344f37b5dae87e\": container with ID starting with ac76e8c198e050c01db771c6243a45a03a683acaa5f8d867e4344f37b5dae87e not found: ID does not exist"
Jul 28 01:51:05.451851 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:05.451821    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" path="/var/lib/kubelet/pods/838f2fa9-b4cc-40db-8d71-c9b8f2d75769/volumes"
Jul 28 01:51:06.702356 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.702313    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"]
Jul 28 01:51:06.702860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.702697    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" containerName="kserve-container"
Jul 28 01:51:06.702860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.702716    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" containerName="kserve-container"
Jul 28 01:51:06.702860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.702737    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" containerName="storage-initializer"
Jul 28 01:51:06.702860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.702747    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" containerName="storage-initializer"
Jul 28 01:51:06.702860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.702757    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" containerName="kube-rbac-proxy"
Jul 28 01:51:06.702860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.702764    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" containerName="kube-rbac-proxy"
Jul 28 01:51:06.702860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.702825    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" containerName="kserve-container"
Jul 28 01:51:06.702860 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.702840    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="838f2fa9-b4cc-40db-8d71-c9b8f2d75769" containerName="kube-rbac-proxy"
Jul 28 01:51:06.707321 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.707299    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:06.710039 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.710006    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Jul 28 01:51:06.710175 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.710012    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 01:51:06.710175 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.710073    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 01:51:06.710175 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.710101    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 01:51:06.711088 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.711071    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Jul 28 01:51:06.711088 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.711083    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 01:51:06.711226 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.711083    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 01:51:06.715495 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.715477    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"]
Jul 28 01:51:06.765664 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.765639    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:06.765813 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.765672    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:06.765813 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.765693    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:06.765813 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.765719    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:06.765813 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.765792    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkkkd\" (UniqueName: \"kubernetes.io/projected/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kube-api-access-wkkkd\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:06.866728 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.866697    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:06.866858 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.866742    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:06.866858 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.866762    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:06.866858 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.866785    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:06.866858 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.866833    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkkkd\" (UniqueName: \"kubernetes.io/projected/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kube-api-access-wkkkd\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:06.867056 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:51:06.866870    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 28 01:51:06.867056 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:51:06.866942    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-proxy-tls podName:f9c6b2bd-b055-4490-ab72-f3c134d3bb2e nodeName:}" failed. No retries permitted until 2026-07-28 01:51:07.366923746 +0000 UTC m=+1514.376442153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr" (UID: "f9c6b2bd-b055-4490-ab72-f3c134d3bb2e") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 28 01:51:06.867152 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.867138    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:06.867422 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.867405    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:06.867466 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.867440    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:06.876167 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:06.876139    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkkkd\" (UniqueName: \"kubernetes.io/projected/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kube-api-access-wkkkd\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:07.370133 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:07.370099    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:07.372643 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:07.372622    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:07.618690 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:07.618650    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:07.749887 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:07.749858    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"]
Jul 28 01:51:07.751856 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:51:07.751821    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c6b2bd_b055_4490_ab72_f3c134d3bb2e.slice/crio-35116caa79b06e4eab9b8c588ffe9151b199da66f28ae4edbd18ee2ac949f750 WatchSource:0}: Error finding container 35116caa79b06e4eab9b8c588ffe9151b199da66f28ae4edbd18ee2ac949f750: Status 404 returned error can't find the container with id 35116caa79b06e4eab9b8c588ffe9151b199da66f28ae4edbd18ee2ac949f750
Jul 28 01:51:07.791783 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:07.791762    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr" event={"ID":"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e","Type":"ContainerStarted","Data":"35116caa79b06e4eab9b8c588ffe9151b199da66f28ae4edbd18ee2ac949f750"}
Jul 28 01:51:08.796126 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:08.796090    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr" event={"ID":"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e","Type":"ContainerStarted","Data":"2e7f9e85fb0ff1ea7ccc32831f0ad9ccfb5f479eb887e8ceacdccc7861d22d3d"}
Jul 28 01:51:17.827687 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:17.827655    2555 generic.go:356] "Generic (PLEG): container finished" podID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" containerID="2e7f9e85fb0ff1ea7ccc32831f0ad9ccfb5f479eb887e8ceacdccc7861d22d3d" exitCode=0
Jul 28 01:51:17.828005 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:17.827692    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr" event={"ID":"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e","Type":"ContainerDied","Data":"2e7f9e85fb0ff1ea7ccc32831f0ad9ccfb5f479eb887e8ceacdccc7861d22d3d"}
Jul 28 01:51:18.833769 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:18.833728    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr" event={"ID":"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e","Type":"ContainerStarted","Data":"8638ae99a0d6c34eade176a2b1aa7df97d483f7a2cf0c197beadce5638f0742b"}
Jul 28 01:51:18.834231 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:18.833800    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr" event={"ID":"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e","Type":"ContainerStarted","Data":"73ac3753ac4d0ac02259656c654e7da55f6c61f1ef05b051b14ae52cebdc7827"}
Jul 28 01:51:18.834231 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:18.834017    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:18.854047 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:18.853999    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr" podStartSLOduration=2.870652072 podStartE2EDuration="12.853983707s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.983331635s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:51:18.852368173 +0000 UTC m=+1525.861886597" watchObservedRunningTime="2026-07-28 01:51:18.853983707 +0000 UTC m=+1525.863502132"
Jul 28 01:51:19.837887 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:19.837854    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:19.839153 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:19.839128    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr" podUID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 01:51:20.843047 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:20.843011    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr" podUID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 01:51:25.848024 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:25.847999    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:25.850262 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:25.848464    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr" podUID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 01:51:35.849557 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:35.849510    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:41.738806 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:41.738770    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"]
Jul 28 01:51:41.739276 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:41.739061    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr" podUID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" containerName="kserve-container" containerID="cri-o://73ac3753ac4d0ac02259656c654e7da55f6c61f1ef05b051b14ae52cebdc7827" gracePeriod=30
Jul 28 01:51:41.739276 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:41.739142    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr" podUID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" containerName="kube-rbac-proxy" containerID="cri-o://8638ae99a0d6c34eade176a2b1aa7df97d483f7a2cf0c197beadce5638f0742b" gracePeriod=30
Jul 28 01:51:41.919711 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:41.919675    2555 generic.go:356] "Generic (PLEG): container finished" podID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" containerID="8638ae99a0d6c34eade176a2b1aa7df97d483f7a2cf0c197beadce5638f0742b" exitCode=2
Jul 28 01:51:41.919857 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:41.919726    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr" event={"ID":"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e","Type":"ContainerDied","Data":"8638ae99a0d6c34eade176a2b1aa7df97d483f7a2cf0c197beadce5638f0742b"}
Jul 28 01:51:42.925567 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:42.925516    2555 generic.go:356] "Generic (PLEG): container finished" podID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" containerID="73ac3753ac4d0ac02259656c654e7da55f6c61f1ef05b051b14ae52cebdc7827" exitCode=0
Jul 28 01:51:42.925567 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:42.925552    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr" event={"ID":"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e","Type":"ContainerDied","Data":"73ac3753ac4d0ac02259656c654e7da55f6c61f1ef05b051b14ae52cebdc7827"}
Jul 28 01:51:43.476158 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.476135    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:43.622165 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.622092    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkkkd\" (UniqueName: \"kubernetes.io/projected/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kube-api-access-wkkkd\") pod \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") "
Jul 28 01:51:43.622165 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.622141    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-cabundle-cert\") pod \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") "
Jul 28 01:51:43.622165 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.622161    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kserve-provision-location\") pod \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") "
Jul 28 01:51:43.622427 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.622189    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-proxy-tls\") pod \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") "
Jul 28 01:51:43.622427 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.622213    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kube-rbac-proxy-sar-config\") pod \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\" (UID: \"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e\") "
Jul 28 01:51:43.622616 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.622578    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" (UID: "f9c6b2bd-b055-4490-ab72-f3c134d3bb2e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:51:43.622716 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.622622    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" (UID: "f9c6b2bd-b055-4490-ab72-f3c134d3bb2e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:51:43.623186 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.623164    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" (UID: "f9c6b2bd-b055-4490-ab72-f3c134d3bb2e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:51:43.624326 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.624303    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" (UID: "f9c6b2bd-b055-4490-ab72-f3c134d3bb2e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:51:43.624407 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.624379    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kube-api-access-wkkkd" (OuterVolumeSpecName: "kube-api-access-wkkkd") pod "f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" (UID: "f9c6b2bd-b055-4490-ab72-f3c134d3bb2e"). InnerVolumeSpecName "kube-api-access-wkkkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:51:43.722589 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.722566    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkkkd\" (UniqueName: \"kubernetes.io/projected/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kube-api-access-wkkkd\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:51:43.722589 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.722586    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:51:43.722738 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.722595    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:51:43.722738 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.722614    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:51:43.722738 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.722622    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:51:43.930127 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.930063    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"
Jul 28 01:51:43.930127 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.930063    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr" event={"ID":"f9c6b2bd-b055-4490-ab72-f3c134d3bb2e","Type":"ContainerDied","Data":"35116caa79b06e4eab9b8c588ffe9151b199da66f28ae4edbd18ee2ac949f750"}
Jul 28 01:51:43.930566 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.930151    2555 scope.go:117] "RemoveContainer" containerID="8638ae99a0d6c34eade176a2b1aa7df97d483f7a2cf0c197beadce5638f0742b"
Jul 28 01:51:43.939156 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.939136    2555 scope.go:117] "RemoveContainer" containerID="73ac3753ac4d0ac02259656c654e7da55f6c61f1ef05b051b14ae52cebdc7827"
Jul 28 01:51:43.946876 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.946853    2555 scope.go:117] "RemoveContainer" containerID="2e7f9e85fb0ff1ea7ccc32831f0ad9ccfb5f479eb887e8ceacdccc7861d22d3d"
Jul 28 01:51:43.951879 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.951834    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"]
Jul 28 01:51:43.953653 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:43.953634    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-nk9zr"]
Jul 28 01:51:45.451263 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:45.451229    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" path="/var/lib/kubelet/pods/f9c6b2bd-b055-4490-ab72-f3c134d3bb2e/volumes"
Jul 28 01:51:46.912278 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.912246    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"]
Jul 28 01:51:46.912646 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.912513    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" containerName="kube-rbac-proxy"
Jul 28 01:51:46.912646 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.912542    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" containerName="kube-rbac-proxy"
Jul 28 01:51:46.912646 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.912559    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" containerName="storage-initializer"
Jul 28 01:51:46.912646 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.912565    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" containerName="storage-initializer"
Jul 28 01:51:46.912646 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.912576    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" containerName="kserve-container"
Jul 28 01:51:46.912646 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.912582    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" containerName="kserve-container"
Jul 28 01:51:46.912646 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.912628    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" containerName="kserve-container"
Jul 28 01:51:46.912646 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.912635    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9c6b2bd-b055-4490-ab72-f3c134d3bb2e" containerName="kube-rbac-proxy"
Jul 28 01:51:46.917090 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.917068    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:46.919404 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.919380    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 01:51:46.919655 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.919636    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 01:51:46.919722 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.919670    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 28 01:51:46.920515 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.920500    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 01:51:46.920515 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.920507    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 01:51:46.920666 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.920546    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 28 01:51:46.920666 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.920546    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 01:51:46.927247 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.927228    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"]
Jul 28 01:51:46.954799 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.954775    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"]
Jul 28 01:51:46.957814 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.957801    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:46.960032 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.960014    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 28 01:51:46.966869 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:46.966848    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"]
Jul 28 01:51:47.045224 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.045193    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-96hqz\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.045350 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.045233    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-96hqz\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.045350 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.045257    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkxc7\" (UniqueName: \"kubernetes.io/projected/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kube-api-access-dkxc7\") pod \"isvc-canary-create-predictor-cb85cf55c-96hqz\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.045420 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.045338    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-96hqz\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.045420 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.045393    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-96hqz\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.146488 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.146462    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-96hqz\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.146621 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.146492    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkxc7\" (UniqueName: \"kubernetes.io/projected/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kube-api-access-dkxc7\") pod \"isvc-canary-create-predictor-cb85cf55c-96hqz\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.146621 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.146515    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-96hqz\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.146621 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.146566    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86c02296-3327-4202-8448-78d92d091b94-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:47.146621 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.146595    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c02296-3327-4202-8448-78d92d091b94-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:47.146843 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.146643    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq2ww\" (UniqueName: \"kubernetes.io/projected/86c02296-3327-4202-8448-78d92d091b94-kube-api-access-vq2ww\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:47.146843 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.146675    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-96hqz\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.146843 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.146699    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86c02296-3327-4202-8448-78d92d091b94-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:47.146843 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.146745    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-96hqz\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.146843 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.146769    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c02296-3327-4202-8448-78d92d091b94-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:47.146992 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:51:47.146875    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 01:51:47.146992 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:51:47.146952    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-proxy-tls podName:3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14 nodeName:}" failed. No retries permitted until 2026-07-28 01:51:47.646923493 +0000 UTC m=+1554.656441915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-proxy-tls") pod "isvc-canary-create-predictor-cb85cf55c-96hqz" (UID: "3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14") : secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 01:51:47.147070 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.146986    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-96hqz\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.147286 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.147266    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-96hqz\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.147322 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.147301    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-96hqz\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.155505 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.155485    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkxc7\" (UniqueName: \"kubernetes.io/projected/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kube-api-access-dkxc7\") pod \"isvc-canary-create-predictor-cb85cf55c-96hqz\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.247704 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.247677    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c02296-3327-4202-8448-78d92d091b94-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:47.247819 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.247723    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86c02296-3327-4202-8448-78d92d091b94-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:47.247819 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.247740    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c02296-3327-4202-8448-78d92d091b94-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:47.247819 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.247767    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq2ww\" (UniqueName: \"kubernetes.io/projected/86c02296-3327-4202-8448-78d92d091b94-kube-api-access-vq2ww\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:47.247819 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.247797    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86c02296-3327-4202-8448-78d92d091b94-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:47.248108 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.248072    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c02296-3327-4202-8448-78d92d091b94-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:47.248421 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.248401    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86c02296-3327-4202-8448-78d92d091b94-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:47.248421 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.248413    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86c02296-3327-4202-8448-78d92d091b94-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:47.250168 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.250147    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c02296-3327-4202-8448-78d92d091b94-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:47.256240 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.256219    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq2ww\" (UniqueName: \"kubernetes.io/projected/86c02296-3327-4202-8448-78d92d091b94-kube-api-access-vq2ww\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:47.268066 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.268043    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:47.395574 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.395553    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"]
Jul 28 01:51:47.398040 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:51:47.398013    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c02296_3327_4202_8448_78d92d091b94.slice/crio-cdb931ed1cca253e662c3c27b219448f85c9a2f8f4849e7ba165ef8b8a1adca6 WatchSource:0}: Error finding container cdb931ed1cca253e662c3c27b219448f85c9a2f8f4849e7ba165ef8b8a1adca6: Status 404 returned error can't find the container with id cdb931ed1cca253e662c3c27b219448f85c9a2f8f4849e7ba165ef8b8a1adca6
Jul 28 01:51:47.651008 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.650930    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-96hqz\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.653374 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.653349    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-96hqz\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.827402 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.827370    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:47.946623 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.946575    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" event={"ID":"86c02296-3327-4202-8448-78d92d091b94","Type":"ContainerStarted","Data":"96c20f40bde476b3a5f24f5d3d8177d6c6f37edee22528b131d27beaf3cfe059"}
Jul 28 01:51:47.946983 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.946660    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" event={"ID":"86c02296-3327-4202-8448-78d92d091b94","Type":"ContainerStarted","Data":"cdb931ed1cca253e662c3c27b219448f85c9a2f8f4849e7ba165ef8b8a1adca6"}
Jul 28 01:51:47.958698 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:47.958558    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"]
Jul 28 01:51:47.961029 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:51:47.961004    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdf9f49_e736_4b4c_a69f_a6b4ff76ae14.slice/crio-393ba27bdb8e7c3065037260efac73b26767d84a4a9e2c838a9b422cc0bc69d9 WatchSource:0}: Error finding container 393ba27bdb8e7c3065037260efac73b26767d84a4a9e2c838a9b422cc0bc69d9: Status 404 returned error can't find the container with id 393ba27bdb8e7c3065037260efac73b26767d84a4a9e2c838a9b422cc0bc69d9
Jul 28 01:51:48.951540 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:48.951485    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" event={"ID":"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14","Type":"ContainerStarted","Data":"943cdad992e5263815108f388f1da692a4eeef216e3ace32e5e8aa04b8075313"}
Jul 28 01:51:48.951934 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:48.951568    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" event={"ID":"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14","Type":"ContainerStarted","Data":"393ba27bdb8e7c3065037260efac73b26767d84a4a9e2c838a9b422cc0bc69d9"}
Jul 28 01:51:51.963003 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:51.962965    2555 generic.go:356] "Generic (PLEG): container finished" podID="86c02296-3327-4202-8448-78d92d091b94" containerID="96c20f40bde476b3a5f24f5d3d8177d6c6f37edee22528b131d27beaf3cfe059" exitCode=0
Jul 28 01:51:51.963449 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:51.963030    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" event={"ID":"86c02296-3327-4202-8448-78d92d091b94","Type":"ContainerDied","Data":"96c20f40bde476b3a5f24f5d3d8177d6c6f37edee22528b131d27beaf3cfe059"}
Jul 28 01:51:51.964711 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:51.964691    2555 generic.go:356] "Generic (PLEG): container finished" podID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerID="943cdad992e5263815108f388f1da692a4eeef216e3ace32e5e8aa04b8075313" exitCode=0
Jul 28 01:51:51.964804 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:51.964779    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" event={"ID":"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14","Type":"ContainerDied","Data":"943cdad992e5263815108f388f1da692a4eeef216e3ace32e5e8aa04b8075313"}
Jul 28 01:51:52.970438 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:52.970406    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" event={"ID":"86c02296-3327-4202-8448-78d92d091b94","Type":"ContainerStarted","Data":"07f1014b1fb2a4dcb74b5d9d392d277ca8487a22542dacf24929a8ad4236b7ce"}
Jul 28 01:51:52.970924 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:52.970470    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" event={"ID":"86c02296-3327-4202-8448-78d92d091b94","Type":"ContainerStarted","Data":"4c24c3174d77a7c35716cbef596dd49c20f8c8dba82de3a2e85f0c767ee71a4b"}
Jul 28 01:51:52.970924 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:52.970760    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:52.970924 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:52.970919    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:52.972385 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:52.972356    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 01:51:52.972768 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:52.972748    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" event={"ID":"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14","Type":"ContainerStarted","Data":"6364d0ba34b9ff884c9743ea1c51bf3f4d29b303707cc44d00c29415bd204b5f"}
Jul 28 01:51:52.972854 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:52.972804    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" event={"ID":"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14","Type":"ContainerStarted","Data":"53341522cb913cf1c6b7513b6db7648c6bc5ef81a410eb4b1d4079a2e23efe3f"}
Jul 28 01:51:52.973009 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:52.972993    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:52.987867 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:52.987824    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" podStartSLOduration=3.408529078 podStartE2EDuration="6.987812609s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.579283531s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:51:52.986590238 +0000 UTC m=+1559.996108680" watchObservedRunningTime="2026-07-28 01:51:52.987812609 +0000 UTC m=+1559.997331034"
Jul 28 01:51:53.004381 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:53.004345    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" podStartSLOduration=3.283492785 podStartE2EDuration="7.004333476s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.720840691s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:51:53.002576459 +0000 UTC m=+1560.012094900" watchObservedRunningTime="2026-07-28 01:51:53.004333476 +0000 UTC m=+1560.013851901"
Jul 28 01:51:53.976542 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:53.976483    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 01:51:53.976542 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:53.976541    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:53.977567 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:53.977512    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 01:51:54.979957 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:54.979918    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 01:51:58.980801 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:58.980770    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:51:58.981246 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:58.981219    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 01:51:59.984402 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:59.984376    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:51:59.984930 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:51:59.984905    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 01:52:08.981687 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:52:08.981643    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 01:52:09.985105 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:52:09.985068    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 01:52:18.981378 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:52:18.981339    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 01:52:19.985459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:52:19.985422    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 01:52:28.982085 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:52:28.982042    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 01:52:29.985081 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:52:29.985039    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 01:52:38.981532 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:52:38.981492    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 01:52:39.984918 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:52:39.984881    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 01:52:48.982012 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:52:48.981934    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 01:52:49.985243 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:52:49.985209    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 01:52:58.981748 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:52:58.981704    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 01:52:59.985404 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:52:59.985365    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 01:53:08.982680 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:08.982650    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:53:09.985766 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:09.985741    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:53:17.026581 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:17.026546    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"]
Jul 28 01:53:17.027052 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:17.026889    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kube-rbac-proxy" containerID="cri-o://07f1014b1fb2a4dcb74b5d9d392d277ca8487a22542dacf24929a8ad4236b7ce" gracePeriod=30
Jul 28 01:53:17.027052 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:17.026883    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kserve-container" containerID="cri-o://4c24c3174d77a7c35716cbef596dd49c20f8c8dba82de3a2e85f0c767ee71a4b" gracePeriod=30
Jul 28 01:53:17.032124 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:17.031930    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"]
Jul 28 01:53:17.032227 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:17.032199    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kserve-container" containerID="cri-o://53341522cb913cf1c6b7513b6db7648c6bc5ef81a410eb4b1d4079a2e23efe3f" gracePeriod=30
Jul 28 01:53:17.032427 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:17.032386    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kube-rbac-proxy" containerID="cri-o://6364d0ba34b9ff884c9743ea1c51bf3f4d29b303707cc44d00c29415bd204b5f" gracePeriod=30
Jul 28 01:53:17.266754 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:17.266697    2555 generic.go:356] "Generic (PLEG): container finished" podID="86c02296-3327-4202-8448-78d92d091b94" containerID="07f1014b1fb2a4dcb74b5d9d392d277ca8487a22542dacf24929a8ad4236b7ce" exitCode=2
Jul 28 01:53:17.266973 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:17.266773    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" event={"ID":"86c02296-3327-4202-8448-78d92d091b94","Type":"ContainerDied","Data":"07f1014b1fb2a4dcb74b5d9d392d277ca8487a22542dacf24929a8ad4236b7ce"}
Jul 28 01:53:17.268870 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:17.268850    2555 generic.go:356] "Generic (PLEG): container finished" podID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerID="6364d0ba34b9ff884c9743ea1c51bf3f4d29b303707cc44d00c29415bd204b5f" exitCode=2
Jul 28 01:53:17.268978 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:17.268918    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" event={"ID":"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14","Type":"ContainerDied","Data":"6364d0ba34b9ff884c9743ea1c51bf3f4d29b303707cc44d00c29415bd204b5f"}
Jul 28 01:53:18.977076 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:18.977033    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 28 01:53:18.981370 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:18.981345    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 01:53:19.980854 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:19.980809    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 28 01:53:19.985147 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:19.985124    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 01:53:21.285542 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.285493    2555 generic.go:356] "Generic (PLEG): container finished" podID="86c02296-3327-4202-8448-78d92d091b94" containerID="4c24c3174d77a7c35716cbef596dd49c20f8c8dba82de3a2e85f0c767ee71a4b" exitCode=0
Jul 28 01:53:21.285900 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.285568    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" event={"ID":"86c02296-3327-4202-8448-78d92d091b94","Type":"ContainerDied","Data":"4c24c3174d77a7c35716cbef596dd49c20f8c8dba82de3a2e85f0c767ee71a4b"}
Jul 28 01:53:21.290424 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.290399    2555 generic.go:356] "Generic (PLEG): container finished" podID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerID="53341522cb913cf1c6b7513b6db7648c6bc5ef81a410eb4b1d4079a2e23efe3f" exitCode=0
Jul 28 01:53:21.290573 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.290467    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" event={"ID":"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14","Type":"ContainerDied","Data":"53341522cb913cf1c6b7513b6db7648c6bc5ef81a410eb4b1d4079a2e23efe3f"}
Jul 28 01:53:21.373897 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.373873    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:53:21.510844 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.510822    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86c02296-3327-4202-8448-78d92d091b94-kube-rbac-proxy-sar-config\") pod \"86c02296-3327-4202-8448-78d92d091b94\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") "
Jul 28 01:53:21.510967 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.510855    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq2ww\" (UniqueName: \"kubernetes.io/projected/86c02296-3327-4202-8448-78d92d091b94-kube-api-access-vq2ww\") pod \"86c02296-3327-4202-8448-78d92d091b94\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") "
Jul 28 01:53:21.510967 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.510879    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86c02296-3327-4202-8448-78d92d091b94-cabundle-cert\") pod \"86c02296-3327-4202-8448-78d92d091b94\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") "
Jul 28 01:53:21.510967 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.510918    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c02296-3327-4202-8448-78d92d091b94-proxy-tls\") pod \"86c02296-3327-4202-8448-78d92d091b94\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") "
Jul 28 01:53:21.510967 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.510945    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c02296-3327-4202-8448-78d92d091b94-kserve-provision-location\") pod \"86c02296-3327-4202-8448-78d92d091b94\" (UID: \"86c02296-3327-4202-8448-78d92d091b94\") "
Jul 28 01:53:21.511249 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.511217    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c02296-3327-4202-8448-78d92d091b94-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86c02296-3327-4202-8448-78d92d091b94" (UID: "86c02296-3327-4202-8448-78d92d091b94"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:53:21.511313 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.511242    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c02296-3327-4202-8448-78d92d091b94-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86c02296-3327-4202-8448-78d92d091b94" (UID: "86c02296-3327-4202-8448-78d92d091b94"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:53:21.511313 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.511298    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c02296-3327-4202-8448-78d92d091b94-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86c02296-3327-4202-8448-78d92d091b94" (UID: "86c02296-3327-4202-8448-78d92d091b94"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:53:21.513155 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.513133    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86c02296-3327-4202-8448-78d92d091b94-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86c02296-3327-4202-8448-78d92d091b94" (UID: "86c02296-3327-4202-8448-78d92d091b94"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:53:21.513264 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.513159    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c02296-3327-4202-8448-78d92d091b94-kube-api-access-vq2ww" (OuterVolumeSpecName: "kube-api-access-vq2ww") pod "86c02296-3327-4202-8448-78d92d091b94" (UID: "86c02296-3327-4202-8448-78d92d091b94"). InnerVolumeSpecName "kube-api-access-vq2ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:53:21.611662 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.611636    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c02296-3327-4202-8448-78d92d091b94-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:53:21.611662 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.611658    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86c02296-3327-4202-8448-78d92d091b94-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:53:21.611811 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.611668    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq2ww\" (UniqueName: \"kubernetes.io/projected/86c02296-3327-4202-8448-78d92d091b94-kube-api-access-vq2ww\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:53:21.611811 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.611678    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86c02296-3327-4202-8448-78d92d091b94-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:53:21.611811 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.611687    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c02296-3327-4202-8448-78d92d091b94-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:53:21.764224 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.764172    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:53:21.914041 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.914011    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-cabundle-cert\") pod \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") "
Jul 28 01:53:21.914199 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.914051    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kube-rbac-proxy-sar-config\") pod \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") "
Jul 28 01:53:21.914199 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.914077    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-proxy-tls\") pod \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") "
Jul 28 01:53:21.914296 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.914199    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkxc7\" (UniqueName: \"kubernetes.io/projected/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kube-api-access-dkxc7\") pod \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") "
Jul 28 01:53:21.914296 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.914243    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kserve-provision-location\") pod \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\" (UID: \"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14\") "
Jul 28 01:53:21.914417 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.914391    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" (UID: "3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:53:21.914477 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.914398    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" (UID: "3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:53:21.914591 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.914569    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" (UID: "3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:53:21.916225 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.916202    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" (UID: "3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:53:21.916350 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:21.916328    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kube-api-access-dkxc7" (OuterVolumeSpecName: "kube-api-access-dkxc7") pod "3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" (UID: "3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14"). InnerVolumeSpecName "kube-api-access-dkxc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:53:22.014874 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.014813    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:53:22.014874 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.014837    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:53:22.015017 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.014875    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:53:22.015017 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.014888    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkxc7\" (UniqueName: \"kubernetes.io/projected/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kube-api-access-dkxc7\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:53:22.015017 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.014900    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:53:22.177255 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177220    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"]
Jul 28 01:53:22.177468 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177457    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kserve-container"
Jul 28 01:53:22.177511 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177471    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kserve-container"
Jul 28 01:53:22.177511 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177482    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="storage-initializer"
Jul 28 01:53:22.177511 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177487    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="storage-initializer"
Jul 28 01:53:22.177511 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177493    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="storage-initializer"
Jul 28 01:53:22.177511 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177498    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="storage-initializer"
Jul 28 01:53:22.177511 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177511    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kube-rbac-proxy"
Jul 28 01:53:22.177715 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177516    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kube-rbac-proxy"
Jul 28 01:53:22.177715 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177538    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kube-rbac-proxy"
Jul 28 01:53:22.177715 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177542    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kube-rbac-proxy"
Jul 28 01:53:22.177715 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177549    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kserve-container"
Jul 28 01:53:22.177715 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177554    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kserve-container"
Jul 28 01:53:22.177715 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177593    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kube-rbac-proxy"
Jul 28 01:53:22.177715 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177599    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c02296-3327-4202-8448-78d92d091b94" containerName="kserve-container"
Jul 28 01:53:22.177715 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177605    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kserve-container"
Jul 28 01:53:22.177715 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.177612    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" containerName="kube-rbac-proxy"
Jul 28 01:53:22.180924 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.180902    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"]
Jul 28 01:53:22.181065 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.181042    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.183453 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.183371    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 28 01:53:22.183582 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.183465    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 01:53:22.183835 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.183816    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:22.186053 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.186033    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 01:53:22.191186 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.191164    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"]
Jul 28 01:53:22.194708 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.194688    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"]
Jul 28 01:53:22.296222 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.296139    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g" event={"ID":"86c02296-3327-4202-8448-78d92d091b94","Type":"ContainerDied","Data":"cdb931ed1cca253e662c3c27b219448f85c9a2f8f4849e7ba165ef8b8a1adca6"}
Jul 28 01:53:22.296222 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.296162    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"
Jul 28 01:53:22.296698 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.296235    2555 scope.go:117] "RemoveContainer" containerID="07f1014b1fb2a4dcb74b5d9d392d277ca8487a22542dacf24929a8ad4236b7ce"
Jul 28 01:53:22.298172 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.298155    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"
Jul 28 01:53:22.298272 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.298166    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz" event={"ID":"3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14","Type":"ContainerDied","Data":"393ba27bdb8e7c3065037260efac73b26767d84a4a9e2c838a9b422cc0bc69d9"}
Jul 28 01:53:22.304571 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.304548    2555 scope.go:117] "RemoveContainer" containerID="4c24c3174d77a7c35716cbef596dd49c20f8c8dba82de3a2e85f0c767ee71a4b"
Jul 28 01:53:22.311876 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.311859    2555 scope.go:117] "RemoveContainer" containerID="96c20f40bde476b3a5f24f5d3d8177d6c6f37edee22528b131d27beaf3cfe059"
Jul 28 01:53:22.316965 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.316941    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-642r7\" (UniqueName: \"kubernetes.io/projected/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kube-api-access-642r7\") pod \"isvc-canary-promote-predictor-d9759ffd5-hvfvr\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.317050 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.316969    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5919e113-7fda-488a-8664-79dad764ebba-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-rt8g4\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:22.317050 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.316987    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a929207-c584-4b9f-8ad7-4e8a7741fba9-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-hvfvr\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.317050 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.317005    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-hvfvr\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.317170 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.317061    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-hvfvr\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.317170 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.317101    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a929207-c584-4b9f-8ad7-4e8a7741fba9-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-hvfvr\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.317170 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.317118    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5919e113-7fda-488a-8664-79dad764ebba-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-rt8g4\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:22.317170 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.317138    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5919e113-7fda-488a-8664-79dad764ebba-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-rt8g4\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:22.317364 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.317203    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgrj7\" (UniqueName: \"kubernetes.io/projected/5919e113-7fda-488a-8664-79dad764ebba-kube-api-access-hgrj7\") pod \"isvc-canary-promote-v2-predictor-86766c4958-rt8g4\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:22.317364 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.317232    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5919e113-7fda-488a-8664-79dad764ebba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-rt8g4\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:22.318637 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.318607    2555 scope.go:117] "RemoveContainer" containerID="6364d0ba34b9ff884c9743ea1c51bf3f4d29b303707cc44d00c29415bd204b5f"
Jul 28 01:53:22.320816 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.320796    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"]
Jul 28 01:53:22.323617 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.323598    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-jmz2g"]
Jul 28 01:53:22.326988 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.326973    2555 scope.go:117] "RemoveContainer" containerID="53341522cb913cf1c6b7513b6db7648c6bc5ef81a410eb4b1d4079a2e23efe3f"
Jul 28 01:53:22.334184 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.334166    2555 scope.go:117] "RemoveContainer" containerID="943cdad992e5263815108f388f1da692a4eeef216e3ace32e5e8aa04b8075313"
Jul 28 01:53:22.334650 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.334626    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"]
Jul 28 01:53:22.337407 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.337388    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-96hqz"]
Jul 28 01:53:22.418422 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.418396    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a929207-c584-4b9f-8ad7-4e8a7741fba9-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-hvfvr\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.418516 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.418427    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5919e113-7fda-488a-8664-79dad764ebba-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-rt8g4\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:22.418516 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.418455    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5919e113-7fda-488a-8664-79dad764ebba-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-rt8g4\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:22.418628 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:53:22.418576    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 28 01:53:22.418628 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.418590    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgrj7\" (UniqueName: \"kubernetes.io/projected/5919e113-7fda-488a-8664-79dad764ebba-kube-api-access-hgrj7\") pod \"isvc-canary-promote-v2-predictor-86766c4958-rt8g4\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:22.418698 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:53:22.418645    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5919e113-7fda-488a-8664-79dad764ebba-proxy-tls podName:5919e113-7fda-488a-8664-79dad764ebba nodeName:}" failed. No retries permitted until 2026-07-28 01:53:22.918616188 +0000 UTC m=+1649.928134590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5919e113-7fda-488a-8664-79dad764ebba-proxy-tls") pod "isvc-canary-promote-v2-predictor-86766c4958-rt8g4" (UID: "5919e113-7fda-488a-8664-79dad764ebba") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 28 01:53:22.418698 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.418673    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5919e113-7fda-488a-8664-79dad764ebba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-rt8g4\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:22.418801 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.418700    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-642r7\" (UniqueName: \"kubernetes.io/projected/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kube-api-access-642r7\") pod \"isvc-canary-promote-predictor-d9759ffd5-hvfvr\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.418801 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.418725    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5919e113-7fda-488a-8664-79dad764ebba-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-rt8g4\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:22.418899 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.418829    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a929207-c584-4b9f-8ad7-4e8a7741fba9-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-hvfvr\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.418899 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.418831    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5919e113-7fda-488a-8664-79dad764ebba-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-rt8g4\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:22.418997 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:53:22.418943    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 28 01:53:22.418997 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.418980    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-hvfvr\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.419087 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:53:22.419025    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a929207-c584-4b9f-8ad7-4e8a7741fba9-proxy-tls podName:1a929207-c584-4b9f-8ad7-4e8a7741fba9 nodeName:}" failed. No retries permitted until 2026-07-28 01:53:22.918993546 +0000 UTC m=+1649.928511962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1a929207-c584-4b9f-8ad7-4e8a7741fba9-proxy-tls") pod "isvc-canary-promote-predictor-d9759ffd5-hvfvr" (UID: "1a929207-c584-4b9f-8ad7-4e8a7741fba9") : secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 28 01:53:22.419087 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.419072    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-hvfvr\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.419173 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.419142    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a929207-c584-4b9f-8ad7-4e8a7741fba9-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-hvfvr\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.419341 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.419324    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-hvfvr\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.419386 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.419372    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5919e113-7fda-488a-8664-79dad764ebba-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-rt8g4\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:22.419431 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.419372    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5919e113-7fda-488a-8664-79dad764ebba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-rt8g4\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:22.419466 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.419450    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-hvfvr\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.427246 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.427219    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-642r7\" (UniqueName: \"kubernetes.io/projected/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kube-api-access-642r7\") pod \"isvc-canary-promote-predictor-d9759ffd5-hvfvr\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.427417 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.427398    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgrj7\" (UniqueName: \"kubernetes.io/projected/5919e113-7fda-488a-8664-79dad764ebba-kube-api-access-hgrj7\") pod \"isvc-canary-promote-v2-predictor-86766c4958-rt8g4\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:22.923073 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.923037    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a929207-c584-4b9f-8ad7-4e8a7741fba9-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-hvfvr\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.923228 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.923089    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5919e113-7fda-488a-8664-79dad764ebba-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-rt8g4\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:22.925614 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.925582    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a929207-c584-4b9f-8ad7-4e8a7741fba9-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-hvfvr\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:22.925741 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:22.925719    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5919e113-7fda-488a-8664-79dad764ebba-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-rt8g4\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:23.093997 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:23.093970    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:23.100631 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:23.100614    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:23.239265 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:23.239243    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"]
Jul 28 01:53:23.241408 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:53:23.241381    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5919e113_7fda_488a_8664_79dad764ebba.slice/crio-cfab933519b78358ca366df5db608e4135b70750393c9d1418a491d9566f2d0b WatchSource:0}: Error finding container cfab933519b78358ca366df5db608e4135b70750393c9d1418a491d9566f2d0b: Status 404 returned error can't find the container with id cfab933519b78358ca366df5db608e4135b70750393c9d1418a491d9566f2d0b
Jul 28 01:53:23.251383 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:23.251354    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"]
Jul 28 01:53:23.254013 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:53:23.253990    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a929207_c584_4b9f_8ad7_4e8a7741fba9.slice/crio-356aa917633626a6cb2909fd05fe5af73c93611b911f709a5bfbb3e14f8470a2 WatchSource:0}: Error finding container 356aa917633626a6cb2909fd05fe5af73c93611b911f709a5bfbb3e14f8470a2: Status 404 returned error can't find the container with id 356aa917633626a6cb2909fd05fe5af73c93611b911f709a5bfbb3e14f8470a2
Jul 28 01:53:23.304682 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:23.304651    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" event={"ID":"5919e113-7fda-488a-8664-79dad764ebba","Type":"ContainerStarted","Data":"cfab933519b78358ca366df5db608e4135b70750393c9d1418a491d9566f2d0b"}
Jul 28 01:53:23.306722 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:23.306607    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" event={"ID":"1a929207-c584-4b9f-8ad7-4e8a7741fba9","Type":"ContainerStarted","Data":"356aa917633626a6cb2909fd05fe5af73c93611b911f709a5bfbb3e14f8470a2"}
Jul 28 01:53:23.452655 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:23.452578    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14" path="/var/lib/kubelet/pods/3fdf9f49-e736-4b4c-a69f-a6b4ff76ae14/volumes"
Jul 28 01:53:23.453089 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:23.453074    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86c02296-3327-4202-8448-78d92d091b94" path="/var/lib/kubelet/pods/86c02296-3327-4202-8448-78d92d091b94/volumes"
Jul 28 01:53:24.311235 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:24.311202    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" event={"ID":"5919e113-7fda-488a-8664-79dad764ebba","Type":"ContainerStarted","Data":"580b79993f94ae505dfba4c75455ac41d434dcca5612f9eb79380dcb6fc67aaa"}
Jul 28 01:53:24.312630 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:24.312608    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" event={"ID":"1a929207-c584-4b9f-8ad7-4e8a7741fba9","Type":"ContainerStarted","Data":"c0b4fef3dbf7eacf671dfda43ee991225be65253cfa8e2fd64720ee5e3827833"}
Jul 28 01:53:27.324592 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:27.324494    2555 generic.go:356] "Generic (PLEG): container finished" podID="5919e113-7fda-488a-8664-79dad764ebba" containerID="580b79993f94ae505dfba4c75455ac41d434dcca5612f9eb79380dcb6fc67aaa" exitCode=0
Jul 28 01:53:27.324592 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:27.324569    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" event={"ID":"5919e113-7fda-488a-8664-79dad764ebba","Type":"ContainerDied","Data":"580b79993f94ae505dfba4c75455ac41d434dcca5612f9eb79380dcb6fc67aaa"}
Jul 28 01:53:27.326093 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:27.326068    2555 generic.go:356] "Generic (PLEG): container finished" podID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerID="c0b4fef3dbf7eacf671dfda43ee991225be65253cfa8e2fd64720ee5e3827833" exitCode=0
Jul 28 01:53:27.326206 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:27.326142    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" event={"ID":"1a929207-c584-4b9f-8ad7-4e8a7741fba9","Type":"ContainerDied","Data":"c0b4fef3dbf7eacf671dfda43ee991225be65253cfa8e2fd64720ee5e3827833"}
Jul 28 01:53:28.331913 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:28.331878    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" event={"ID":"5919e113-7fda-488a-8664-79dad764ebba","Type":"ContainerStarted","Data":"6cf686320862da89ee0c35d59ed5a6211b765413a15bcdd70f3d929b67afdbc9"}
Jul 28 01:53:28.332377 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:28.331944    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" event={"ID":"5919e113-7fda-488a-8664-79dad764ebba","Type":"ContainerStarted","Data":"b2e7fc08e53925aae4694277f24f62b130b1844bf8f88745512aa0120be01e00"}
Jul 28 01:53:28.332377 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:28.332154    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:28.333890 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:28.333869    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" event={"ID":"1a929207-c584-4b9f-8ad7-4e8a7741fba9","Type":"ContainerStarted","Data":"579544d4c530ac71dec9f0ced95819ab026196795326a87069387fb1b963a8db"}
Jul 28 01:53:28.333981 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:28.333912    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" event={"ID":"1a929207-c584-4b9f-8ad7-4e8a7741fba9","Type":"ContainerStarted","Data":"135487241c3b1e40b7cb6d0761fb0d598fa7da1459e1f9df5705d8f2f63c6352"}
Jul 28 01:53:28.334216 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:28.334201    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:28.334253 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:28.334241    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:28.335735 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:28.335709    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:53:28.349538 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:28.349484    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" podStartSLOduration=2.570554506 podStartE2EDuration="6.349471355s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.778916849s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:53:28.348975644 +0000 UTC m=+1655.358494070" watchObservedRunningTime="2026-07-28 01:53:28.349471355 +0000 UTC m=+1655.358989780"
Jul 28 01:53:28.365960 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:28.365912    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" podStartSLOduration=2.670927503 podStartE2EDuration="6.365895148s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.694967645s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:53:28.365139585 +0000 UTC m=+1655.374658011" watchObservedRunningTime="2026-07-28 01:53:28.365895148 +0000 UTC m=+1655.375413586"
Jul 28 01:53:29.338072 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:29.338038    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:29.338072 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:29.338061    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:53:29.339116 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:29.339094    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 01:53:30.341537 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:30.341485    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 01:53:34.342590 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:34.342565    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:53:34.343038 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:34.343013    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:53:35.347037 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:35.347010    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:53:35.347656 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:35.347631    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 01:53:44.343161 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:44.343119    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:53:45.348075 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:45.348034    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 01:53:54.343016 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:54.342976    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:53:55.348551 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:53:55.348499    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 01:54:04.343743 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:04.343706    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:54:05.347697 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:05.347652    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 01:54:14.343789 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:14.343711    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:54:15.347889 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:15.347848    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 01:54:24.344000 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:24.343954    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:54:25.347750 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:25.347714    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 01:54:34.343503 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:34.343466    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:54:35.348556 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:35.348500    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 01:54:44.344123 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:44.344095    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:54:45.348475 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:45.348447    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:54:52.237539 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:52.237483    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"]
Jul 28 01:54:52.237961 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:52.237770    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kserve-container" containerID="cri-o://135487241c3b1e40b7cb6d0761fb0d598fa7da1459e1f9df5705d8f2f63c6352" gracePeriod=30
Jul 28 01:54:52.237961 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:52.237802    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kube-rbac-proxy" containerID="cri-o://579544d4c530ac71dec9f0ced95819ab026196795326a87069387fb1b963a8db" gracePeriod=30
Jul 28 01:54:52.623165 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:52.623077    2555 generic.go:356] "Generic (PLEG): container finished" podID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerID="579544d4c530ac71dec9f0ced95819ab026196795326a87069387fb1b963a8db" exitCode=2
Jul 28 01:54:52.623165 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:52.623138    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" event={"ID":"1a929207-c584-4b9f-8ad7-4e8a7741fba9","Type":"ContainerDied","Data":"579544d4c530ac71dec9f0ced95819ab026196795326a87069387fb1b963a8db"}
Jul 28 01:54:54.338465 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:54.338422    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 01:54:54.343771 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:54.343743    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:54:56.376013 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.375989    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:54:56.460491 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.460425    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-642r7\" (UniqueName: \"kubernetes.io/projected/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kube-api-access-642r7\") pod \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") "
Jul 28 01:54:56.460491 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.460473    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kserve-provision-location\") pod \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") "
Jul 28 01:54:56.460656 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.460596    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a929207-c584-4b9f-8ad7-4e8a7741fba9-proxy-tls\") pod \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") "
Jul 28 01:54:56.460656 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.460626    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a929207-c584-4b9f-8ad7-4e8a7741fba9-cabundle-cert\") pod \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") "
Jul 28 01:54:56.460764 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.460676    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kube-rbac-proxy-sar-config\") pod \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\" (UID: \"1a929207-c584-4b9f-8ad7-4e8a7741fba9\") "
Jul 28 01:54:56.460816 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.460789    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a929207-c584-4b9f-8ad7-4e8a7741fba9" (UID: "1a929207-c584-4b9f-8ad7-4e8a7741fba9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:54:56.460920 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.460894    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:54:56.461045 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.460977    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a929207-c584-4b9f-8ad7-4e8a7741fba9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a929207-c584-4b9f-8ad7-4e8a7741fba9" (UID: "1a929207-c584-4b9f-8ad7-4e8a7741fba9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:54:56.461115 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.461058    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a929207-c584-4b9f-8ad7-4e8a7741fba9" (UID: "1a929207-c584-4b9f-8ad7-4e8a7741fba9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:54:56.462728 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.462708    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kube-api-access-642r7" (OuterVolumeSpecName: "kube-api-access-642r7") pod "1a929207-c584-4b9f-8ad7-4e8a7741fba9" (UID: "1a929207-c584-4b9f-8ad7-4e8a7741fba9"). InnerVolumeSpecName "kube-api-access-642r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:54:56.462787 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.462740    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a929207-c584-4b9f-8ad7-4e8a7741fba9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a929207-c584-4b9f-8ad7-4e8a7741fba9" (UID: "1a929207-c584-4b9f-8ad7-4e8a7741fba9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:54:56.561947 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.561922    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a929207-c584-4b9f-8ad7-4e8a7741fba9-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:54:56.561947 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.561944    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a929207-c584-4b9f-8ad7-4e8a7741fba9-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:54:56.562079 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.561954    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:54:56.562079 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.561963    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-642r7\" (UniqueName: \"kubernetes.io/projected/1a929207-c584-4b9f-8ad7-4e8a7741fba9-kube-api-access-642r7\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:54:56.640255 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.640228    2555 generic.go:356] "Generic (PLEG): container finished" podID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerID="135487241c3b1e40b7cb6d0761fb0d598fa7da1459e1f9df5705d8f2f63c6352" exitCode=0
Jul 28 01:54:56.640354 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.640288    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"
Jul 28 01:54:56.640354 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.640305    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" event={"ID":"1a929207-c584-4b9f-8ad7-4e8a7741fba9","Type":"ContainerDied","Data":"135487241c3b1e40b7cb6d0761fb0d598fa7da1459e1f9df5705d8f2f63c6352"}
Jul 28 01:54:56.640431 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.640365    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr" event={"ID":"1a929207-c584-4b9f-8ad7-4e8a7741fba9","Type":"ContainerDied","Data":"356aa917633626a6cb2909fd05fe5af73c93611b911f709a5bfbb3e14f8470a2"}
Jul 28 01:54:56.640431 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.640391    2555 scope.go:117] "RemoveContainer" containerID="579544d4c530ac71dec9f0ced95819ab026196795326a87069387fb1b963a8db"
Jul 28 01:54:56.648877 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.648858    2555 scope.go:117] "RemoveContainer" containerID="135487241c3b1e40b7cb6d0761fb0d598fa7da1459e1f9df5705d8f2f63c6352"
Jul 28 01:54:56.655763 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.655744    2555 scope.go:117] "RemoveContainer" containerID="c0b4fef3dbf7eacf671dfda43ee991225be65253cfa8e2fd64720ee5e3827833"
Jul 28 01:54:56.661366 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.661345    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"]
Jul 28 01:54:56.663592 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.663569    2555 scope.go:117] "RemoveContainer" containerID="579544d4c530ac71dec9f0ced95819ab026196795326a87069387fb1b963a8db"
Jul 28 01:54:56.663815 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:54:56.663794    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"579544d4c530ac71dec9f0ced95819ab026196795326a87069387fb1b963a8db\": container with ID starting with 579544d4c530ac71dec9f0ced95819ab026196795326a87069387fb1b963a8db not found: ID does not exist" containerID="579544d4c530ac71dec9f0ced95819ab026196795326a87069387fb1b963a8db"
Jul 28 01:54:56.663872 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.663827    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"579544d4c530ac71dec9f0ced95819ab026196795326a87069387fb1b963a8db"} err="failed to get container status \"579544d4c530ac71dec9f0ced95819ab026196795326a87069387fb1b963a8db\": rpc error: code = NotFound desc = could not find container \"579544d4c530ac71dec9f0ced95819ab026196795326a87069387fb1b963a8db\": container with ID starting with 579544d4c530ac71dec9f0ced95819ab026196795326a87069387fb1b963a8db not found: ID does not exist"
Jul 28 01:54:56.663872 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.663845    2555 scope.go:117] "RemoveContainer" containerID="135487241c3b1e40b7cb6d0761fb0d598fa7da1459e1f9df5705d8f2f63c6352"
Jul 28 01:54:56.664111 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:54:56.664095    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"135487241c3b1e40b7cb6d0761fb0d598fa7da1459e1f9df5705d8f2f63c6352\": container with ID starting with 135487241c3b1e40b7cb6d0761fb0d598fa7da1459e1f9df5705d8f2f63c6352 not found: ID does not exist" containerID="135487241c3b1e40b7cb6d0761fb0d598fa7da1459e1f9df5705d8f2f63c6352"
Jul 28 01:54:56.664146 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.664116    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"135487241c3b1e40b7cb6d0761fb0d598fa7da1459e1f9df5705d8f2f63c6352"} err="failed to get container status \"135487241c3b1e40b7cb6d0761fb0d598fa7da1459e1f9df5705d8f2f63c6352\": rpc error: code = NotFound desc = could not find container \"135487241c3b1e40b7cb6d0761fb0d598fa7da1459e1f9df5705d8f2f63c6352\": container with ID starting with 135487241c3b1e40b7cb6d0761fb0d598fa7da1459e1f9df5705d8f2f63c6352 not found: ID does not exist"
Jul 28 01:54:56.664146 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.664135    2555 scope.go:117] "RemoveContainer" containerID="c0b4fef3dbf7eacf671dfda43ee991225be65253cfa8e2fd64720ee5e3827833"
Jul 28 01:54:56.664348 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:54:56.664332    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b4fef3dbf7eacf671dfda43ee991225be65253cfa8e2fd64720ee5e3827833\": container with ID starting with c0b4fef3dbf7eacf671dfda43ee991225be65253cfa8e2fd64720ee5e3827833 not found: ID does not exist" containerID="c0b4fef3dbf7eacf671dfda43ee991225be65253cfa8e2fd64720ee5e3827833"
Jul 28 01:54:56.664396 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.664360    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b4fef3dbf7eacf671dfda43ee991225be65253cfa8e2fd64720ee5e3827833"} err="failed to get container status \"c0b4fef3dbf7eacf671dfda43ee991225be65253cfa8e2fd64720ee5e3827833\": rpc error: code = NotFound desc = could not find container \"c0b4fef3dbf7eacf671dfda43ee991225be65253cfa8e2fd64720ee5e3827833\": container with ID starting with c0b4fef3dbf7eacf671dfda43ee991225be65253cfa8e2fd64720ee5e3827833 not found: ID does not exist"
Jul 28 01:54:56.664970 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:56.664952    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-hvfvr"]
Jul 28 01:54:57.451661 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:54:57.451627    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" path="/var/lib/kubelet/pods/1a929207-c584-4b9f-8ad7-4e8a7741fba9/volumes"
Jul 28 01:56:42.486160 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:42.486129    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"]
Jul 28 01:56:42.488416 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:42.486388    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kserve-container" containerID="cri-o://b2e7fc08e53925aae4694277f24f62b130b1844bf8f88745512aa0120be01e00" gracePeriod=30
Jul 28 01:56:42.488416 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:42.486409    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kube-rbac-proxy" containerID="cri-o://6cf686320862da89ee0c35d59ed5a6211b765413a15bcdd70f3d929b67afdbc9" gracePeriod=30
Jul 28 01:56:42.990182 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:42.990151    2555 generic.go:356] "Generic (PLEG): container finished" podID="5919e113-7fda-488a-8664-79dad764ebba" containerID="6cf686320862da89ee0c35d59ed5a6211b765413a15bcdd70f3d929b67afdbc9" exitCode=2
Jul 28 01:56:42.990332 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:42.990226    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" event={"ID":"5919e113-7fda-488a-8664-79dad764ebba","Type":"ContainerDied","Data":"6cf686320862da89ee0c35d59ed5a6211b765413a15bcdd70f3d929b67afdbc9"}
Jul 28 01:56:45.342744 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:45.342702    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 28 01:56:45.348127 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:45.348094    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 01:56:46.429450 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:46.429426    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:56:46.472729 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:46.472703    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5919e113-7fda-488a-8664-79dad764ebba-kserve-provision-location\") pod \"5919e113-7fda-488a-8664-79dad764ebba\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") "
Jul 28 01:56:46.472842 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:46.472747    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5919e113-7fda-488a-8664-79dad764ebba-cabundle-cert\") pod \"5919e113-7fda-488a-8664-79dad764ebba\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") "
Jul 28 01:56:46.472842 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:46.472778    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5919e113-7fda-488a-8664-79dad764ebba-proxy-tls\") pod \"5919e113-7fda-488a-8664-79dad764ebba\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") "
Jul 28 01:56:46.472923 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:46.472893    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgrj7\" (UniqueName: \"kubernetes.io/projected/5919e113-7fda-488a-8664-79dad764ebba-kube-api-access-hgrj7\") pod \"5919e113-7fda-488a-8664-79dad764ebba\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") "
Jul 28 01:56:46.472975 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:46.472923    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5919e113-7fda-488a-8664-79dad764ebba-kube-rbac-proxy-sar-config\") pod \"5919e113-7fda-488a-8664-79dad764ebba\" (UID: \"5919e113-7fda-488a-8664-79dad764ebba\") "
Jul 28 01:56:46.473071 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:46.473045    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5919e113-7fda-488a-8664-79dad764ebba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5919e113-7fda-488a-8664-79dad764ebba" (UID: "5919e113-7fda-488a-8664-79dad764ebba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:56:46.473134 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:46.473071    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5919e113-7fda-488a-8664-79dad764ebba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5919e113-7fda-488a-8664-79dad764ebba" (UID: "5919e113-7fda-488a-8664-79dad764ebba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:56:46.473134 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:46.473130    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5919e113-7fda-488a-8664-79dad764ebba-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:56:46.473237 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:46.473141    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5919e113-7fda-488a-8664-79dad764ebba-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:56:46.473307 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:46.473286    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5919e113-7fda-488a-8664-79dad764ebba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5919e113-7fda-488a-8664-79dad764ebba" (UID: "5919e113-7fda-488a-8664-79dad764ebba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:56:46.474852 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:46.474833    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5919e113-7fda-488a-8664-79dad764ebba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5919e113-7fda-488a-8664-79dad764ebba" (UID: "5919e113-7fda-488a-8664-79dad764ebba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:56:46.474973 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:46.474957    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5919e113-7fda-488a-8664-79dad764ebba-kube-api-access-hgrj7" (OuterVolumeSpecName: "kube-api-access-hgrj7") pod "5919e113-7fda-488a-8664-79dad764ebba" (UID: "5919e113-7fda-488a-8664-79dad764ebba"). InnerVolumeSpecName "kube-api-access-hgrj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:56:46.574079 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:46.574026    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5919e113-7fda-488a-8664-79dad764ebba-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:56:46.574079 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:46.574047    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgrj7\" (UniqueName: \"kubernetes.io/projected/5919e113-7fda-488a-8664-79dad764ebba-kube-api-access-hgrj7\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:56:46.574079 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:46.574063    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5919e113-7fda-488a-8664-79dad764ebba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:56:47.004534 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.004491    2555 generic.go:356] "Generic (PLEG): container finished" podID="5919e113-7fda-488a-8664-79dad764ebba" containerID="b2e7fc08e53925aae4694277f24f62b130b1844bf8f88745512aa0120be01e00" exitCode=0
Jul 28 01:56:47.004694 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.004558    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"
Jul 28 01:56:47.004694 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.004582    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" event={"ID":"5919e113-7fda-488a-8664-79dad764ebba","Type":"ContainerDied","Data":"b2e7fc08e53925aae4694277f24f62b130b1844bf8f88745512aa0120be01e00"}
Jul 28 01:56:47.004694 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.004655    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4" event={"ID":"5919e113-7fda-488a-8664-79dad764ebba","Type":"ContainerDied","Data":"cfab933519b78358ca366df5db608e4135b70750393c9d1418a491d9566f2d0b"}
Jul 28 01:56:47.004694 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.004691    2555 scope.go:117] "RemoveContainer" containerID="6cf686320862da89ee0c35d59ed5a6211b765413a15bcdd70f3d929b67afdbc9"
Jul 28 01:56:47.017001 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.012895    2555 scope.go:117] "RemoveContainer" containerID="b2e7fc08e53925aae4694277f24f62b130b1844bf8f88745512aa0120be01e00"
Jul 28 01:56:47.024144 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.024115    2555 scope.go:117] "RemoveContainer" containerID="580b79993f94ae505dfba4c75455ac41d434dcca5612f9eb79380dcb6fc67aaa"
Jul 28 01:56:47.027949 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.027928    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"]
Jul 28 01:56:47.031370 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.031354    2555 scope.go:117] "RemoveContainer" containerID="6cf686320862da89ee0c35d59ed5a6211b765413a15bcdd70f3d929b67afdbc9"
Jul 28 01:56:47.031653 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:56:47.031634    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf686320862da89ee0c35d59ed5a6211b765413a15bcdd70f3d929b67afdbc9\": container with ID starting with 6cf686320862da89ee0c35d59ed5a6211b765413a15bcdd70f3d929b67afdbc9 not found: ID does not exist" containerID="6cf686320862da89ee0c35d59ed5a6211b765413a15bcdd70f3d929b67afdbc9"
Jul 28 01:56:47.031704 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.031661    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf686320862da89ee0c35d59ed5a6211b765413a15bcdd70f3d929b67afdbc9"} err="failed to get container status \"6cf686320862da89ee0c35d59ed5a6211b765413a15bcdd70f3d929b67afdbc9\": rpc error: code = NotFound desc = could not find container \"6cf686320862da89ee0c35d59ed5a6211b765413a15bcdd70f3d929b67afdbc9\": container with ID starting with 6cf686320862da89ee0c35d59ed5a6211b765413a15bcdd70f3d929b67afdbc9 not found: ID does not exist"
Jul 28 01:56:47.031704 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.031678    2555 scope.go:117] "RemoveContainer" containerID="b2e7fc08e53925aae4694277f24f62b130b1844bf8f88745512aa0120be01e00"
Jul 28 01:56:47.031898 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:56:47.031884    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e7fc08e53925aae4694277f24f62b130b1844bf8f88745512aa0120be01e00\": container with ID starting with b2e7fc08e53925aae4694277f24f62b130b1844bf8f88745512aa0120be01e00 not found: ID does not exist" containerID="b2e7fc08e53925aae4694277f24f62b130b1844bf8f88745512aa0120be01e00"
Jul 28 01:56:47.031941 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.031902    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e7fc08e53925aae4694277f24f62b130b1844bf8f88745512aa0120be01e00"} err="failed to get container status \"b2e7fc08e53925aae4694277f24f62b130b1844bf8f88745512aa0120be01e00\": rpc error: code = NotFound desc = could not find container \"b2e7fc08e53925aae4694277f24f62b130b1844bf8f88745512aa0120be01e00\": container with ID starting with b2e7fc08e53925aae4694277f24f62b130b1844bf8f88745512aa0120be01e00 not found: ID does not exist"
Jul 28 01:56:47.031941 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.031915    2555 scope.go:117] "RemoveContainer" containerID="580b79993f94ae505dfba4c75455ac41d434dcca5612f9eb79380dcb6fc67aaa"
Jul 28 01:56:47.032132 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:56:47.032112    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580b79993f94ae505dfba4c75455ac41d434dcca5612f9eb79380dcb6fc67aaa\": container with ID starting with 580b79993f94ae505dfba4c75455ac41d434dcca5612f9eb79380dcb6fc67aaa not found: ID does not exist" containerID="580b79993f94ae505dfba4c75455ac41d434dcca5612f9eb79380dcb6fc67aaa"
Jul 28 01:56:47.032179 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.032138    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580b79993f94ae505dfba4c75455ac41d434dcca5612f9eb79380dcb6fc67aaa"} err="failed to get container status \"580b79993f94ae505dfba4c75455ac41d434dcca5612f9eb79380dcb6fc67aaa\": rpc error: code = NotFound desc = could not find container \"580b79993f94ae505dfba4c75455ac41d434dcca5612f9eb79380dcb6fc67aaa\": container with ID starting with 580b79993f94ae505dfba4c75455ac41d434dcca5612f9eb79380dcb6fc67aaa not found: ID does not exist"
Jul 28 01:56:47.033820 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.033800    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-rt8g4"]
Jul 28 01:56:47.451474 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.451414    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5919e113-7fda-488a-8664-79dad764ebba" path="/var/lib/kubelet/pods/5919e113-7fda-488a-8664-79dad764ebba/volumes"
Jul 28 01:56:47.639446 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639416    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"]
Jul 28 01:56:47.639659 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639648    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kserve-container"
Jul 28 01:56:47.639709 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639661    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kserve-container"
Jul 28 01:56:47.639709 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639669    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="storage-initializer"
Jul 28 01:56:47.639709 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639675    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="storage-initializer"
Jul 28 01:56:47.639709 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639685    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="storage-initializer"
Jul 28 01:56:47.639709 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639691    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="storage-initializer"
Jul 28 01:56:47.639709 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639703    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kserve-container"
Jul 28 01:56:47.639709 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639708    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kserve-container"
Jul 28 01:56:47.639937 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639714    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kube-rbac-proxy"
Jul 28 01:56:47.639937 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639720    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kube-rbac-proxy"
Jul 28 01:56:47.639937 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639725    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kube-rbac-proxy"
Jul 28 01:56:47.639937 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639730    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kube-rbac-proxy"
Jul 28 01:56:47.639937 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639768    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kube-rbac-proxy"
Jul 28 01:56:47.639937 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639775    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a929207-c584-4b9f-8ad7-4e8a7741fba9" containerName="kserve-container"
Jul 28 01:56:47.639937 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639781    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kserve-container"
Jul 28 01:56:47.639937 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.639788    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5919e113-7fda-488a-8664-79dad764ebba" containerName="kube-rbac-proxy"
Jul 28 01:56:47.643930 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.643908    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"]
Jul 28 01:56:47.644063 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.644046    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:47.646361 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.646339    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 01:56:47.646620 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.646601    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 01:56:47.646708 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.646602    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 01:56:47.646708 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.646609    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 01:56:47.647329 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.647308    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 01:56:47.647418 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.647363    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 01:56:47.647418 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.647367    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 01:56:47.647665 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.647646    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:47.651124 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.650958    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 28 01:56:47.653100 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.653081    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"]
Jul 28 01:56:47.659108 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.659085    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"]
Jul 28 01:56:47.781410 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.781377    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wf2g\" (UniqueName: \"kubernetes.io/projected/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kube-api-access-5wf2g\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:47.781567 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.781420    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad9c41e5-0817-4e9f-b870-5c3081314708-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-glcxz\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:47.781567 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.781445    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:47.781567 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.781483    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad9c41e5-0817-4e9f-b870-5c3081314708-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-glcxz\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:47.781567 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.781517    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad9c41e5-0817-4e9f-b870-5c3081314708-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-glcxz\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:47.781703 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.781569    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6jv\" (UniqueName: \"kubernetes.io/projected/ad9c41e5-0817-4e9f-b870-5c3081314708-kube-api-access-ck6jv\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-glcxz\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:47.781703 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.781597    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:47.781703 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.781648    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-glcxz\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:47.781703 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.781678    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:47.781703 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.781697    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:47.882819 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.882789    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6jv\" (UniqueName: \"kubernetes.io/projected/ad9c41e5-0817-4e9f-b870-5c3081314708-kube-api-access-ck6jv\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-glcxz\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:47.882922 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.882827    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:47.882922 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.882846    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-glcxz\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:47.882922 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.882895    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:47.883043 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.882923    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:47.883043 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.882951    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wf2g\" (UniqueName: \"kubernetes.io/projected/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kube-api-access-5wf2g\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:47.883043 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.882973    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad9c41e5-0817-4e9f-b870-5c3081314708-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-glcxz\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:47.883043 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.882996    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:47.883043 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:56:47.882924    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 01:56:47.883043 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.883020    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad9c41e5-0817-4e9f-b870-5c3081314708-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-glcxz\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:47.883043 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:56:47.883026    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 01:56:47.883373 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:56:47.883081    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls podName:ad9c41e5-0817-4e9f-b870-5c3081314708 nodeName:}" failed. No retries permitted until 2026-07-28 01:56:48.383059464 +0000 UTC m=+1855.392577867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls") pod "isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" (UID: "ad9c41e5-0817-4e9f-b870-5c3081314708") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 01:56:47.883373 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.883103    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad9c41e5-0817-4e9f-b870-5c3081314708-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-glcxz\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:47.883373 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:56:47.883165    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-proxy-tls podName:d1bfe636-5b16-4ee0-8458-a1ab8de8c022 nodeName:}" failed. No retries permitted until 2026-07-28 01:56:48.383148183 +0000 UTC m=+1855.392666586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-proxy-tls") pod "isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" (UID: "d1bfe636-5b16-4ee0-8458-a1ab8de8c022") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 01:56:47.883571 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.883378    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:47.883571 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.883483    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad9c41e5-0817-4e9f-b870-5c3081314708-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-glcxz\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:47.883691 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.883616    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:47.883691 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.883665    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:47.883691 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.883681    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad9c41e5-0817-4e9f-b870-5c3081314708-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-glcxz\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:47.883837 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.883768    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad9c41e5-0817-4e9f-b870-5c3081314708-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-glcxz\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:47.898047 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.898021    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wf2g\" (UniqueName: \"kubernetes.io/projected/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kube-api-access-5wf2g\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:47.898160 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:47.898056    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6jv\" (UniqueName: \"kubernetes.io/projected/ad9c41e5-0817-4e9f-b870-5c3081314708-kube-api-access-ck6jv\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-glcxz\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:48.387816 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:48.387788    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-glcxz\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:48.387964 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:48.387821    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:48.390240 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:48.390209    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:48.390382 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:48.390294    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-glcxz\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:48.555327 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:48.555301    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:48.562484 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:48.562468    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:48.694492 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:48.694462    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"]
Jul 28 01:56:48.697447 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:56:48.697420    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1bfe636_5b16_4ee0_8458_a1ab8de8c022.slice/crio-b09a5f70c8f1997413d1d39239f47d829ca9ec4cda4b23b0eecacc9719e57afd WatchSource:0}: Error finding container b09a5f70c8f1997413d1d39239f47d829ca9ec4cda4b23b0eecacc9719e57afd: Status 404 returned error can't find the container with id b09a5f70c8f1997413d1d39239f47d829ca9ec4cda4b23b0eecacc9719e57afd
Jul 28 01:56:48.712969 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:48.712943    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"]
Jul 28 01:56:48.714464 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:56:48.714429    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9c41e5_0817_4e9f_b870_5c3081314708.slice/crio-3d0271c55807d7e99e775e8862125ab38d51312a00603f6d7c577352ed50788e WatchSource:0}: Error finding container 3d0271c55807d7e99e775e8862125ab38d51312a00603f6d7c577352ed50788e: Status 404 returned error can't find the container with id 3d0271c55807d7e99e775e8862125ab38d51312a00603f6d7c577352ed50788e
Jul 28 01:56:49.016438 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:49.016400    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" event={"ID":"ad9c41e5-0817-4e9f-b870-5c3081314708","Type":"ContainerStarted","Data":"4c8b6cc78ecc632d594a8d285a3aff1383f7cb6631eee06c280c17a899ee061b"}
Jul 28 01:56:49.016641 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:49.016475    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" event={"ID":"ad9c41e5-0817-4e9f-b870-5c3081314708","Type":"ContainerStarted","Data":"3d0271c55807d7e99e775e8862125ab38d51312a00603f6d7c577352ed50788e"}
Jul 28 01:56:49.018024 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:49.017998    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" event={"ID":"d1bfe636-5b16-4ee0-8458-a1ab8de8c022","Type":"ContainerStarted","Data":"92ab7df6cbf9e62032bef0934a97fdd6b1d78c15cc64f199a1077ae2c4bcc674"}
Jul 28 01:56:49.018111 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:49.018048    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" event={"ID":"d1bfe636-5b16-4ee0-8458-a1ab8de8c022","Type":"ContainerStarted","Data":"b09a5f70c8f1997413d1d39239f47d829ca9ec4cda4b23b0eecacc9719e57afd"}
Jul 28 01:56:53.033272 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:53.033243    2555 generic.go:356] "Generic (PLEG): container finished" podID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerID="92ab7df6cbf9e62032bef0934a97fdd6b1d78c15cc64f199a1077ae2c4bcc674" exitCode=0
Jul 28 01:56:53.033719 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:53.033323    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" event={"ID":"d1bfe636-5b16-4ee0-8458-a1ab8de8c022","Type":"ContainerDied","Data":"92ab7df6cbf9e62032bef0934a97fdd6b1d78c15cc64f199a1077ae2c4bcc674"}
Jul 28 01:56:53.034841 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:53.034821    2555 generic.go:356] "Generic (PLEG): container finished" podID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerID="4c8b6cc78ecc632d594a8d285a3aff1383f7cb6631eee06c280c17a899ee061b" exitCode=0
Jul 28 01:56:53.034936 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:53.034858    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" event={"ID":"ad9c41e5-0817-4e9f-b870-5c3081314708","Type":"ContainerDied","Data":"4c8b6cc78ecc632d594a8d285a3aff1383f7cb6631eee06c280c17a899ee061b"}
Jul 28 01:56:54.039820 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:54.039785    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" event={"ID":"d1bfe636-5b16-4ee0-8458-a1ab8de8c022","Type":"ContainerStarted","Data":"bc3d82e9dbc4bdad5cd303cc224f5d280df4ef98344a6ca6c63477fd520fcc23"}
Jul 28 01:56:54.040271 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:54.039853    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" event={"ID":"d1bfe636-5b16-4ee0-8458-a1ab8de8c022","Type":"ContainerStarted","Data":"d987a822ffe409cf60f5ade766420143db42bad982c69285738cd6978ca56e91"}
Jul 28 01:56:54.040271 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:54.040062    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:54.041944 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:54.041924    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" event={"ID":"ad9c41e5-0817-4e9f-b870-5c3081314708","Type":"ContainerStarted","Data":"38217faf1903452eb6c9ae745b4fbdb836b98c760fe4958ce8dac6a1cb32d141"}
Jul 28 01:56:54.042059 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:54.041964    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" event={"ID":"ad9c41e5-0817-4e9f-b870-5c3081314708","Type":"ContainerStarted","Data":"2a422f72969135947c4f5a3520868489f206fcf972523231e72aac788080cfa3"}
Jul 28 01:56:54.042150 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:54.042130    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:54.042254 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:54.042172    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:56:54.043507 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:54.043485    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 01:56:54.057834 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:54.057787    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" podStartSLOduration=3.323450114 podStartE2EDuration="7.057773263s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.734323149s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:56:54.0572571 +0000 UTC m=+1861.066775524" watchObservedRunningTime="2026-07-28 01:56:54.057773263 +0000 UTC m=+1861.067291684"
Jul 28 01:56:54.078363 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:54.078318    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" podStartSLOduration=3.314950523 podStartE2EDuration="7.078305758s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.763355235s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:56:54.076893914 +0000 UTC m=+1861.086412337" watchObservedRunningTime="2026-07-28 01:56:54.078305758 +0000 UTC m=+1861.087824181"
Jul 28 01:56:55.045912 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:55.045863    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 01:56:55.046309 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:55.046104    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:56:55.047238 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:55.047213    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 01:56:56.049597 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:56:56.049561    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 01:57:00.051514 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:57:00.051483    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:57:00.052115 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:57:00.052086    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 01:57:01.054214 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:57:01.054187    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:57:01.054735 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:57:01.054710    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 01:57:10.052768 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:57:10.052726    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 01:57:11.055540 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:57:11.055485    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 01:57:20.052248 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:57:20.052165    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 01:57:21.055723 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:57:21.055687    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 01:57:30.052870 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:57:30.052825    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 01:57:31.055684 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:57:31.055642    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 01:57:40.052112 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:57:40.052068    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 01:57:41.055662 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:57:41.055622    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 01:57:50.052474 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:57:50.052435    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 01:57:51.054995 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:57:51.054951    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 01:58:00.052765 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:00.052720    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 01:58:01.055666 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:01.055636    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:58:07.451308 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:07.451280    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:58:07.743477 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:07.743441    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"]
Jul 28 01:58:08.350541 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:58:08.350482    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 01:58:08.350724 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:58:08.350585    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls podName:ad9c41e5-0817-4e9f-b870-5c3081314708 nodeName:}" failed. No retries permitted until 2026-07-28 01:58:08.850565018 +0000 UTC m=+1935.860083421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls") pod "isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" (UID: "ad9c41e5-0817-4e9f-b870-5c3081314708") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 01:58:08.853803 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:58:08.853767    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 01:58:08.854267 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:58:08.853856    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls podName:ad9c41e5-0817-4e9f-b870-5c3081314708 nodeName:}" failed. No retries permitted until 2026-07-28 01:58:09.853836865 +0000 UTC m=+1936.863355268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls") pod "isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" (UID: "ad9c41e5-0817-4e9f-b870-5c3081314708") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 01:58:09.304495 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:09.304432    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kserve-container" containerID="cri-o://2a422f72969135947c4f5a3520868489f206fcf972523231e72aac788080cfa3" gracePeriod=30
Jul 28 01:58:09.304697 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:09.304459    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kube-rbac-proxy" containerID="cri-o://38217faf1903452eb6c9ae745b4fbdb836b98c760fe4958ce8dac6a1cb32d141" gracePeriod=30
Jul 28 01:58:09.862883 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:58:09.862835    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 01:58:09.863292 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:58:09.862924    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls podName:ad9c41e5-0817-4e9f-b870-5c3081314708 nodeName:}" failed. No retries permitted until 2026-07-28 01:58:11.862905097 +0000 UTC m=+1938.872423503 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls") pod "isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" (UID: "ad9c41e5-0817-4e9f-b870-5c3081314708") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 01:58:10.046454 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:10.046411    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 28 01:58:10.310342 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:10.310307    2555 generic.go:356] "Generic (PLEG): container finished" podID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerID="38217faf1903452eb6c9ae745b4fbdb836b98c760fe4958ce8dac6a1cb32d141" exitCode=2
Jul 28 01:58:10.310513 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:10.310352    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" event={"ID":"ad9c41e5-0817-4e9f-b870-5c3081314708","Type":"ContainerDied","Data":"38217faf1903452eb6c9ae745b4fbdb836b98c760fe4958ce8dac6a1cb32d141"}
Jul 28 01:58:11.882998 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:58:11.882964    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 01:58:11.883453 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:58:11.883042    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls podName:ad9c41e5-0817-4e9f-b870-5c3081314708 nodeName:}" failed. No retries permitted until 2026-07-28 01:58:15.883023514 +0000 UTC m=+1942.892541917 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls") pod "isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" (UID: "ad9c41e5-0817-4e9f-b870-5c3081314708") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 01:58:13.751715 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:13.751689    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:58:13.899502 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:13.899419    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck6jv\" (UniqueName: \"kubernetes.io/projected/ad9c41e5-0817-4e9f-b870-5c3081314708-kube-api-access-ck6jv\") pod \"ad9c41e5-0817-4e9f-b870-5c3081314708\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") "
Jul 28 01:58:13.899502 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:13.899458    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad9c41e5-0817-4e9f-b870-5c3081314708-kube-rbac-proxy-sar-config\") pod \"ad9c41e5-0817-4e9f-b870-5c3081314708\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") "
Jul 28 01:58:13.899502 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:13.899493    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls\") pod \"ad9c41e5-0817-4e9f-b870-5c3081314708\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") "
Jul 28 01:58:13.899809 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:13.899610    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad9c41e5-0817-4e9f-b870-5c3081314708-cabundle-cert\") pod \"ad9c41e5-0817-4e9f-b870-5c3081314708\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") "
Jul 28 01:58:13.899809 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:13.899678    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad9c41e5-0817-4e9f-b870-5c3081314708-kserve-provision-location\") pod \"ad9c41e5-0817-4e9f-b870-5c3081314708\" (UID: \"ad9c41e5-0817-4e9f-b870-5c3081314708\") "
Jul 28 01:58:13.899898 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:13.899869    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad9c41e5-0817-4e9f-b870-5c3081314708-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad9c41e5-0817-4e9f-b870-5c3081314708" (UID: "ad9c41e5-0817-4e9f-b870-5c3081314708"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:58:13.899994 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:13.899961    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9c41e5-0817-4e9f-b870-5c3081314708-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad9c41e5-0817-4e9f-b870-5c3081314708" (UID: "ad9c41e5-0817-4e9f-b870-5c3081314708"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:58:13.899994 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:13.899992    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad9c41e5-0817-4e9f-b870-5c3081314708-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad9c41e5-0817-4e9f-b870-5c3081314708" (UID: "ad9c41e5-0817-4e9f-b870-5c3081314708"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:58:13.901808 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:13.901779    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad9c41e5-0817-4e9f-b870-5c3081314708" (UID: "ad9c41e5-0817-4e9f-b870-5c3081314708"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:58:13.901896 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:13.901860    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9c41e5-0817-4e9f-b870-5c3081314708-kube-api-access-ck6jv" (OuterVolumeSpecName: "kube-api-access-ck6jv") pod "ad9c41e5-0817-4e9f-b870-5c3081314708" (UID: "ad9c41e5-0817-4e9f-b870-5c3081314708"). InnerVolumeSpecName "kube-api-access-ck6jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:58:14.000711 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.000681    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad9c41e5-0817-4e9f-b870-5c3081314708-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:58:14.000711 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.000706    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad9c41e5-0817-4e9f-b870-5c3081314708-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:58:14.000711 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.000716    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck6jv\" (UniqueName: \"kubernetes.io/projected/ad9c41e5-0817-4e9f-b870-5c3081314708-kube-api-access-ck6jv\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:58:14.000933 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.000724    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad9c41e5-0817-4e9f-b870-5c3081314708-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:58:14.000933 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.000733    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad9c41e5-0817-4e9f-b870-5c3081314708-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:58:14.324314 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.324278    2555 generic.go:356] "Generic (PLEG): container finished" podID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerID="2a422f72969135947c4f5a3520868489f206fcf972523231e72aac788080cfa3" exitCode=0
Jul 28 01:58:14.324493 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.324326    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" event={"ID":"ad9c41e5-0817-4e9f-b870-5c3081314708","Type":"ContainerDied","Data":"2a422f72969135947c4f5a3520868489f206fcf972523231e72aac788080cfa3"}
Jul 28 01:58:14.324493 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.324351    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"
Jul 28 01:58:14.324493 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.324377    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz" event={"ID":"ad9c41e5-0817-4e9f-b870-5c3081314708","Type":"ContainerDied","Data":"3d0271c55807d7e99e775e8862125ab38d51312a00603f6d7c577352ed50788e"}
Jul 28 01:58:14.324493 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.324405    2555 scope.go:117] "RemoveContainer" containerID="38217faf1903452eb6c9ae745b4fbdb836b98c760fe4958ce8dac6a1cb32d141"
Jul 28 01:58:14.332880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.332861    2555 scope.go:117] "RemoveContainer" containerID="2a422f72969135947c4f5a3520868489f206fcf972523231e72aac788080cfa3"
Jul 28 01:58:14.339961 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.339940    2555 scope.go:117] "RemoveContainer" containerID="4c8b6cc78ecc632d594a8d285a3aff1383f7cb6631eee06c280c17a899ee061b"
Jul 28 01:58:14.345769 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.345747    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"]
Jul 28 01:58:14.347136 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.347117    2555 scope.go:117] "RemoveContainer" containerID="38217faf1903452eb6c9ae745b4fbdb836b98c760fe4958ce8dac6a1cb32d141"
Jul 28 01:58:14.347386 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:58:14.347366    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38217faf1903452eb6c9ae745b4fbdb836b98c760fe4958ce8dac6a1cb32d141\": container with ID starting with 38217faf1903452eb6c9ae745b4fbdb836b98c760fe4958ce8dac6a1cb32d141 not found: ID does not exist" containerID="38217faf1903452eb6c9ae745b4fbdb836b98c760fe4958ce8dac6a1cb32d141"
Jul 28 01:58:14.347465 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.347398    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38217faf1903452eb6c9ae745b4fbdb836b98c760fe4958ce8dac6a1cb32d141"} err="failed to get container status \"38217faf1903452eb6c9ae745b4fbdb836b98c760fe4958ce8dac6a1cb32d141\": rpc error: code = NotFound desc = could not find container \"38217faf1903452eb6c9ae745b4fbdb836b98c760fe4958ce8dac6a1cb32d141\": container with ID starting with 38217faf1903452eb6c9ae745b4fbdb836b98c760fe4958ce8dac6a1cb32d141 not found: ID does not exist"
Jul 28 01:58:14.347465 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.347424    2555 scope.go:117] "RemoveContainer" containerID="2a422f72969135947c4f5a3520868489f206fcf972523231e72aac788080cfa3"
Jul 28 01:58:14.347700 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:58:14.347679    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a422f72969135947c4f5a3520868489f206fcf972523231e72aac788080cfa3\": container with ID starting with 2a422f72969135947c4f5a3520868489f206fcf972523231e72aac788080cfa3 not found: ID does not exist" containerID="2a422f72969135947c4f5a3520868489f206fcf972523231e72aac788080cfa3"
Jul 28 01:58:14.347741 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.347706    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a422f72969135947c4f5a3520868489f206fcf972523231e72aac788080cfa3"} err="failed to get container status \"2a422f72969135947c4f5a3520868489f206fcf972523231e72aac788080cfa3\": rpc error: code = NotFound desc = could not find container \"2a422f72969135947c4f5a3520868489f206fcf972523231e72aac788080cfa3\": container with ID starting with 2a422f72969135947c4f5a3520868489f206fcf972523231e72aac788080cfa3 not found: ID does not exist"
Jul 28 01:58:14.347741 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.347722    2555 scope.go:117] "RemoveContainer" containerID="4c8b6cc78ecc632d594a8d285a3aff1383f7cb6631eee06c280c17a899ee061b"
Jul 28 01:58:14.347969 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:58:14.347952    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c8b6cc78ecc632d594a8d285a3aff1383f7cb6631eee06c280c17a899ee061b\": container with ID starting with 4c8b6cc78ecc632d594a8d285a3aff1383f7cb6631eee06c280c17a899ee061b not found: ID does not exist" containerID="4c8b6cc78ecc632d594a8d285a3aff1383f7cb6631eee06c280c17a899ee061b"
Jul 28 01:58:14.348022 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.347976    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8b6cc78ecc632d594a8d285a3aff1383f7cb6631eee06c280c17a899ee061b"} err="failed to get container status \"4c8b6cc78ecc632d594a8d285a3aff1383f7cb6631eee06c280c17a899ee061b\": rpc error: code = NotFound desc = could not find container \"4c8b6cc78ecc632d594a8d285a3aff1383f7cb6631eee06c280c17a899ee061b\": container with ID starting with 4c8b6cc78ecc632d594a8d285a3aff1383f7cb6631eee06c280c17a899ee061b not found: ID does not exist"
Jul 28 01:58:14.350110 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:14.350091    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-glcxz"]
Jul 28 01:58:15.451064 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:58:15.451028    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" path="/var/lib/kubelet/pods/ad9c41e5-0817-4e9f-b870-5c3081314708/volumes"
Jul 28 01:59:37.913383 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:37.913352    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"]
Jul 28 01:59:37.913911 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:37.913636    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kserve-container" containerID="cri-o://d987a822ffe409cf60f5ade766420143db42bad982c69285738cd6978ca56e91" gracePeriod=30
Jul 28 01:59:37.913911 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:37.913658    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kube-rbac-proxy" containerID="cri-o://bc3d82e9dbc4bdad5cd303cc224f5d280df4ef98344a6ca6c63477fd520fcc23" gracePeriod=30
Jul 28 01:59:38.606820 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:38.606788    2555 generic.go:356] "Generic (PLEG): container finished" podID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerID="bc3d82e9dbc4bdad5cd303cc224f5d280df4ef98344a6ca6c63477fd520fcc23" exitCode=2
Jul 28 01:59:38.606990 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:38.606860    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" event={"ID":"d1bfe636-5b16-4ee0-8458-a1ab8de8c022","Type":"ContainerDied","Data":"bc3d82e9dbc4bdad5cd303cc224f5d280df4ef98344a6ca6c63477fd520fcc23"}
Jul 28 01:59:41.049959 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:41.049916    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 28 01:59:41.055249 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:41.055227    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 01:59:42.570312 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.570289    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:59:42.617946 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.617879    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-cabundle-cert\") pod \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") "
Jul 28 01:59:42.617946 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.617922    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-proxy-tls\") pod \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") "
Jul 28 01:59:42.618120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.617958    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wf2g\" (UniqueName: \"kubernetes.io/projected/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kube-api-access-5wf2g\") pod \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") "
Jul 28 01:59:42.618120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.618075    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kube-rbac-proxy-sar-config\") pod \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") "
Jul 28 01:59:42.618120 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.618116    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kserve-provision-location\") pod \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\" (UID: \"d1bfe636-5b16-4ee0-8458-a1ab8de8c022\") "
Jul 28 01:59:42.618288 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.618257    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d1bfe636-5b16-4ee0-8458-a1ab8de8c022" (UID: "d1bfe636-5b16-4ee0-8458-a1ab8de8c022"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:59:42.618349 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.618335    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:59:42.618554 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.618486    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d1bfe636-5b16-4ee0-8458-a1ab8de8c022" (UID: "d1bfe636-5b16-4ee0-8458-a1ab8de8c022"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:59:42.618667 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.618568    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d1bfe636-5b16-4ee0-8458-a1ab8de8c022" (UID: "d1bfe636-5b16-4ee0-8458-a1ab8de8c022"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:59:42.620713 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.620688    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1bfe636-5b16-4ee0-8458-a1ab8de8c022" (UID: "d1bfe636-5b16-4ee0-8458-a1ab8de8c022"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:59:42.620814 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.620725    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kube-api-access-5wf2g" (OuterVolumeSpecName: "kube-api-access-5wf2g") pod "d1bfe636-5b16-4ee0-8458-a1ab8de8c022" (UID: "d1bfe636-5b16-4ee0-8458-a1ab8de8c022"). InnerVolumeSpecName "kube-api-access-5wf2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:59:42.622510 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.622489    2555 generic.go:356] "Generic (PLEG): container finished" podID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerID="d987a822ffe409cf60f5ade766420143db42bad982c69285738cd6978ca56e91" exitCode=0
Jul 28 01:59:42.622620 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.622560    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" event={"ID":"d1bfe636-5b16-4ee0-8458-a1ab8de8c022","Type":"ContainerDied","Data":"d987a822ffe409cf60f5ade766420143db42bad982c69285738cd6978ca56e91"}
Jul 28 01:59:42.622620 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.622604    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"
Jul 28 01:59:42.622696 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.622621    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s" event={"ID":"d1bfe636-5b16-4ee0-8458-a1ab8de8c022","Type":"ContainerDied","Data":"b09a5f70c8f1997413d1d39239f47d829ca9ec4cda4b23b0eecacc9719e57afd"}
Jul 28 01:59:42.622696 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.622649    2555 scope.go:117] "RemoveContainer" containerID="bc3d82e9dbc4bdad5cd303cc224f5d280df4ef98344a6ca6c63477fd520fcc23"
Jul 28 01:59:42.635684 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.635664    2555 scope.go:117] "RemoveContainer" containerID="d987a822ffe409cf60f5ade766420143db42bad982c69285738cd6978ca56e91"
Jul 28 01:59:42.642633 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.642599    2555 scope.go:117] "RemoveContainer" containerID="92ab7df6cbf9e62032bef0934a97fdd6b1d78c15cc64f199a1077ae2c4bcc674"
Jul 28 01:59:42.646419 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.646398    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"]
Jul 28 01:59:42.649614 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.649592    2555 scope.go:117] "RemoveContainer" containerID="bc3d82e9dbc4bdad5cd303cc224f5d280df4ef98344a6ca6c63477fd520fcc23"
Jul 28 01:59:42.649870 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:59:42.649853    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3d82e9dbc4bdad5cd303cc224f5d280df4ef98344a6ca6c63477fd520fcc23\": container with ID starting with bc3d82e9dbc4bdad5cd303cc224f5d280df4ef98344a6ca6c63477fd520fcc23 not found: ID does not exist" containerID="bc3d82e9dbc4bdad5cd303cc224f5d280df4ef98344a6ca6c63477fd520fcc23"
Jul 28 01:59:42.649924 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.649877    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3d82e9dbc4bdad5cd303cc224f5d280df4ef98344a6ca6c63477fd520fcc23"} err="failed to get container status \"bc3d82e9dbc4bdad5cd303cc224f5d280df4ef98344a6ca6c63477fd520fcc23\": rpc error: code = NotFound desc = could not find container \"bc3d82e9dbc4bdad5cd303cc224f5d280df4ef98344a6ca6c63477fd520fcc23\": container with ID starting with bc3d82e9dbc4bdad5cd303cc224f5d280df4ef98344a6ca6c63477fd520fcc23 not found: ID does not exist"
Jul 28 01:59:42.649924 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.649901    2555 scope.go:117] "RemoveContainer" containerID="d987a822ffe409cf60f5ade766420143db42bad982c69285738cd6978ca56e91"
Jul 28 01:59:42.650091 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:59:42.650077    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d987a822ffe409cf60f5ade766420143db42bad982c69285738cd6978ca56e91\": container with ID starting with d987a822ffe409cf60f5ade766420143db42bad982c69285738cd6978ca56e91 not found: ID does not exist" containerID="d987a822ffe409cf60f5ade766420143db42bad982c69285738cd6978ca56e91"
Jul 28 01:59:42.650134 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.650094    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d987a822ffe409cf60f5ade766420143db42bad982c69285738cd6978ca56e91"} err="failed to get container status \"d987a822ffe409cf60f5ade766420143db42bad982c69285738cd6978ca56e91\": rpc error: code = NotFound desc = could not find container \"d987a822ffe409cf60f5ade766420143db42bad982c69285738cd6978ca56e91\": container with ID starting with d987a822ffe409cf60f5ade766420143db42bad982c69285738cd6978ca56e91 not found: ID does not exist"
Jul 28 01:59:42.650134 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.650107    2555 scope.go:117] "RemoveContainer" containerID="92ab7df6cbf9e62032bef0934a97fdd6b1d78c15cc64f199a1077ae2c4bcc674"
Jul 28 01:59:42.650345 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:59:42.650326    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ab7df6cbf9e62032bef0934a97fdd6b1d78c15cc64f199a1077ae2c4bcc674\": container with ID starting with 92ab7df6cbf9e62032bef0934a97fdd6b1d78c15cc64f199a1077ae2c4bcc674 not found: ID does not exist" containerID="92ab7df6cbf9e62032bef0934a97fdd6b1d78c15cc64f199a1077ae2c4bcc674"
Jul 28 01:59:42.650394 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.650347    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ab7df6cbf9e62032bef0934a97fdd6b1d78c15cc64f199a1077ae2c4bcc674"} err="failed to get container status \"92ab7df6cbf9e62032bef0934a97fdd6b1d78c15cc64f199a1077ae2c4bcc674\": rpc error: code = NotFound desc = could not find container \"92ab7df6cbf9e62032bef0934a97fdd6b1d78c15cc64f199a1077ae2c4bcc674\": container with ID starting with 92ab7df6cbf9e62032bef0934a97fdd6b1d78c15cc64f199a1077ae2c4bcc674 not found: ID does not exist"
Jul 28 01:59:42.652695 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.652669    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-5p46s"]
Jul 28 01:59:42.719305 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.719279    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wf2g\" (UniqueName: \"kubernetes.io/projected/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kube-api-access-5wf2g\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:59:42.719305 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.719301    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:59:42.719442 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.719313    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:59:42.719442 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.719322    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1bfe636-5b16-4ee0-8458-a1ab8de8c022-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 01:59:42.990882 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.990848    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"]
Jul 28 01:59:42.991185 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.991168    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kube-rbac-proxy"
Jul 28 01:59:42.991276 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.991188    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kube-rbac-proxy"
Jul 28 01:59:42.991276 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.991209    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="storage-initializer"
Jul 28 01:59:42.991276 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.991218    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="storage-initializer"
Jul 28 01:59:42.991276 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.991233    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kserve-container"
Jul 28 01:59:42.991276 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.991242    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kserve-container"
Jul 28 01:59:42.991276 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.991254    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kube-rbac-proxy"
Jul 28 01:59:42.991276 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.991263    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kube-rbac-proxy"
Jul 28 01:59:42.991276 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.991274    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="storage-initializer"
Jul 28 01:59:42.991700 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.991282    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="storage-initializer"
Jul 28 01:59:42.991700 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.991296    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kserve-container"
Jul 28 01:59:42.991700 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.991304    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kserve-container"
Jul 28 01:59:42.991700 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.991366    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kube-rbac-proxy"
Jul 28 01:59:42.991700 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.991378    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad9c41e5-0817-4e9f-b870-5c3081314708" containerName="kserve-container"
Jul 28 01:59:42.991700 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.991388    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kube-rbac-proxy"
Jul 28 01:59:42.991700 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.991398    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" containerName="kserve-container"
Jul 28 01:59:42.996022 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.996001    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"]
Jul 28 01:59:42.996146 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.996133    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:42.998377 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.998356    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 28 01:59:42.998685 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.998669    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 01:59:42.999094 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.999077    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 01:59:42.999182 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.999079    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 01:59:42.999182 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.999080    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 01:59:42.999342 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.999324    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:42.999461 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.999377    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 01:59:42.999626 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:42.999610    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 01:59:43.001934 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.001913    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 28 01:59:43.003846 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.003825    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"]
Jul 28 01:59:43.009698 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.009679    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"]
Jul 28 01:59:43.021072 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.021041    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-csttb\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:43.021180 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.021077    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq75n\" (UniqueName: \"kubernetes.io/projected/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kube-api-access-cq75n\") pod \"isvc-canary-stop-predictor-86dc9fdb65-csttb\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:43.021180 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.021101    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-csttb\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:43.021180 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.021168    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-csttb\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:43.021305 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.021262    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-csttb\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:43.122232 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.122201    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-csttb\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:43.122389 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.122236    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz5w5\" (UniqueName: \"kubernetes.io/projected/8f77be49-28a3-4860-a7a3-766b28efb55f-kube-api-access-lz5w5\") pod \"isvc-canary-stop-v2-predictor-77f778d864-f92zr\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:43.122389 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.122264    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f77be49-28a3-4860-a7a3-766b28efb55f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-f92zr\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:43.122389 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.122289    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-csttb\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:43.122389 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.122314    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq75n\" (UniqueName: \"kubernetes.io/projected/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kube-api-access-cq75n\") pod \"isvc-canary-stop-predictor-86dc9fdb65-csttb\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:43.122389 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.122333    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f77be49-28a3-4860-a7a3-766b28efb55f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-f92zr\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:43.122389 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.122350    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f77be49-28a3-4860-a7a3-766b28efb55f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-f92zr\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:43.122389 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.122382    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-csttb\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:43.122741 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.122444    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-csttb\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:43.122741 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.122494    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f77be49-28a3-4860-a7a3-766b28efb55f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-f92zr\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:43.122741 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:59:43.122576    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 28 01:59:43.122741 ip-10-0-133-123 kubenswrapper[2555]: E0728 01:59:43.122667    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-proxy-tls podName:de2a26bb-3bf1-4e1b-b1b0-3d332cce2488 nodeName:}" failed. No retries permitted until 2026-07-28 01:59:43.622645974 +0000 UTC m=+2030.632164390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-proxy-tls") pod "isvc-canary-stop-predictor-86dc9fdb65-csttb" (UID: "de2a26bb-3bf1-4e1b-b1b0-3d332cce2488") : secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 28 01:59:43.122880 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.122862    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-csttb\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:43.122915 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.122895    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-csttb\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:43.122997 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.122980    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-csttb\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:43.129787 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.129762    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq75n\" (UniqueName: \"kubernetes.io/projected/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kube-api-access-cq75n\") pod \"isvc-canary-stop-predictor-86dc9fdb65-csttb\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:43.222915 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.222884    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f77be49-28a3-4860-a7a3-766b28efb55f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-f92zr\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:43.222915 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.222917    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f77be49-28a3-4860-a7a3-766b28efb55f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-f92zr\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:43.223134 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.222935    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f77be49-28a3-4860-a7a3-766b28efb55f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-f92zr\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:43.223134 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.222968    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f77be49-28a3-4860-a7a3-766b28efb55f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-f92zr\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:43.223134 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.223086    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz5w5\" (UniqueName: \"kubernetes.io/projected/8f77be49-28a3-4860-a7a3-766b28efb55f-kube-api-access-lz5w5\") pod \"isvc-canary-stop-v2-predictor-77f778d864-f92zr\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:43.223364 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.223344    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f77be49-28a3-4860-a7a3-766b28efb55f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-f92zr\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:43.223671 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.223649    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f77be49-28a3-4860-a7a3-766b28efb55f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-f92zr\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:43.223753 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.223681    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f77be49-28a3-4860-a7a3-766b28efb55f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-f92zr\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:43.225443 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.225424    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f77be49-28a3-4860-a7a3-766b28efb55f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-f92zr\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:43.230170 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.230154    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz5w5\" (UniqueName: \"kubernetes.io/projected/8f77be49-28a3-4860-a7a3-766b28efb55f-kube-api-access-lz5w5\") pod \"isvc-canary-stop-v2-predictor-77f778d864-f92zr\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:43.314459 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.314391    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:43.443137 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.443107    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"]
Jul 28 01:59:43.446299 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:59:43.446265    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f77be49_28a3_4860_a7a3_766b28efb55f.slice/crio-8083d5db2d3bd27138ace0800f7fc3e7f3a429dd7170b98eab6a57b4220c5dc5 WatchSource:0}: Error finding container 8083d5db2d3bd27138ace0800f7fc3e7f3a429dd7170b98eab6a57b4220c5dc5: Status 404 returned error can't find the container with id 8083d5db2d3bd27138ace0800f7fc3e7f3a429dd7170b98eab6a57b4220c5dc5
Jul 28 01:59:43.452213 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.452188    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1bfe636-5b16-4ee0-8458-a1ab8de8c022" path="/var/lib/kubelet/pods/d1bfe636-5b16-4ee0-8458-a1ab8de8c022/volumes"
Jul 28 01:59:43.626127 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.626038    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-csttb\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:43.628859 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.628835    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-csttb\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:43.629518 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.629486    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" event={"ID":"8f77be49-28a3-4860-a7a3-766b28efb55f","Type":"ContainerStarted","Data":"923f2bc98ef62bc74cddbec3ee045d82269e559acf05f0f4cc46c99fe8d55e8a"}
Jul 28 01:59:43.629657 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.629578    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" event={"ID":"8f77be49-28a3-4860-a7a3-766b28efb55f","Type":"ContainerStarted","Data":"8083d5db2d3bd27138ace0800f7fc3e7f3a429dd7170b98eab6a57b4220c5dc5"}
Jul 28 01:59:43.907624 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:43.907548    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:44.059585 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:44.059553    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"]
Jul 28 01:59:44.062276 ip-10-0-133-123 kubenswrapper[2555]: W0728 01:59:44.062252    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2a26bb_3bf1_4e1b_b1b0_3d332cce2488.slice/crio-66b51fbcae1b7dbb90093950e8c3132a2d02414fa0b2d5818dd28bb311bbad47 WatchSource:0}: Error finding container 66b51fbcae1b7dbb90093950e8c3132a2d02414fa0b2d5818dd28bb311bbad47: Status 404 returned error can't find the container with id 66b51fbcae1b7dbb90093950e8c3132a2d02414fa0b2d5818dd28bb311bbad47
Jul 28 01:59:44.634264 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:44.634228    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" event={"ID":"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488","Type":"ContainerStarted","Data":"5ebda43edcc2001b4e54b0dcfd967947f05d3a329490fdde28a7bea1528d4b3d"}
Jul 28 01:59:44.634688 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:44.634284    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" event={"ID":"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488","Type":"ContainerStarted","Data":"66b51fbcae1b7dbb90093950e8c3132a2d02414fa0b2d5818dd28bb311bbad47"}
Jul 28 01:59:47.645399 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:47.645367    2555 generic.go:356] "Generic (PLEG): container finished" podID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerID="923f2bc98ef62bc74cddbec3ee045d82269e559acf05f0f4cc46c99fe8d55e8a" exitCode=0
Jul 28 01:59:47.645792 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:47.645405    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" event={"ID":"8f77be49-28a3-4860-a7a3-766b28efb55f","Type":"ContainerDied","Data":"923f2bc98ef62bc74cddbec3ee045d82269e559acf05f0f4cc46c99fe8d55e8a"}
Jul 28 01:59:48.652073 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:48.652036    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" event={"ID":"8f77be49-28a3-4860-a7a3-766b28efb55f","Type":"ContainerStarted","Data":"ad7525e8bd441d50aa88a8704ef2a5a001f4b7a96ea1a469d5c0a8e544407ad7"}
Jul 28 01:59:48.652566 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:48.652107    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" event={"ID":"8f77be49-28a3-4860-a7a3-766b28efb55f","Type":"ContainerStarted","Data":"9cff9ddae8297e66d578541bbf4cd654307f911245cddcb948512043b5c4f8bb"}
Jul 28 01:59:48.652566 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:48.652430    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:48.652566 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:48.652488    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:48.653621 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:48.653590    2555 generic.go:356] "Generic (PLEG): container finished" podID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerID="5ebda43edcc2001b4e54b0dcfd967947f05d3a329490fdde28a7bea1528d4b3d" exitCode=0
Jul 28 01:59:48.653737 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:48.653642    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" event={"ID":"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488","Type":"ContainerDied","Data":"5ebda43edcc2001b4e54b0dcfd967947f05d3a329490fdde28a7bea1528d4b3d"}
Jul 28 01:59:48.653858 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:48.653833    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 01:59:48.670238 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:48.670197    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" podStartSLOduration=3.058885347 podStartE2EDuration="6.670183297s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.61129795s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:59:48.669387409 +0000 UTC m=+2035.678905857" watchObservedRunningTime="2026-07-28 01:59:48.670183297 +0000 UTC m=+2035.679701751"
Jul 28 01:59:49.659015 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:49.658972    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" event={"ID":"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488","Type":"ContainerStarted","Data":"17760b756e68e9cf4abbf486179b6faefa110359884273a22674eef302c304b6"}
Jul 28 01:59:49.659370 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:49.659054    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" event={"ID":"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488","Type":"ContainerStarted","Data":"da15d3041634333051498a4334b0d83dec224c8e82d14b964ce73db4b0e31f03"}
Jul 28 01:59:49.659465 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:49.659364    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:49.659465 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:49.659435    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 01:59:49.679749 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:49.679705    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" podStartSLOduration=3.848406458 podStartE2EDuration="7.679691059s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.831284601s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:59:49.679101646 +0000 UTC m=+2036.688620071" watchObservedRunningTime="2026-07-28 01:59:49.679691059 +0000 UTC m=+2036.689209483"
Jul 28 01:59:50.664649 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:50.664615    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:50.666049 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:50.666019    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 01:59:51.668028 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:51.667980    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 01:59:54.665318 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:54.665288    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 01:59:54.665774 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:54.665734    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 01:59:56.672854 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:56.672828    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 01:59:56.673285 ip-10-0-133-123 kubenswrapper[2555]: I0728 01:59:56.673258    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 02:00:04.666754 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:00:04.666706    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 02:00:06.673900 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:00:06.673860    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 02:00:14.666751 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:00:14.666669    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 02:00:16.673332 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:00:16.673294    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 02:00:24.666543 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:00:24.666485    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 02:00:26.674174 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:00:26.674138    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 02:00:34.666058 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:00:34.666015    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 02:00:36.673604 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:00:36.673568    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 02:00:44.666006 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:00:44.665953    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 02:00:46.673698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:00:46.673658    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 02:00:54.666037 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:00:54.665992    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 02:00:56.673354 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:00:56.673321    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 02:01:03.451345 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:03.451317    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 02:01:04.667252 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:04.667219    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 02:01:13.017874 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:13.017843    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"]
Jul 28 02:01:13.018325 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:13.018131    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kube-rbac-proxy" containerID="cri-o://17760b756e68e9cf4abbf486179b6faefa110359884273a22674eef302c304b6" gracePeriod=30
Jul 28 02:01:13.018325 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:13.018131    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kserve-container" containerID="cri-o://da15d3041634333051498a4334b0d83dec224c8e82d14b964ce73db4b0e31f03" gracePeriod=30
Jul 28 02:01:13.048238 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:13.048213    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"]
Jul 28 02:01:13.048456 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:13.048430    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kserve-container" containerID="cri-o://9cff9ddae8297e66d578541bbf4cd654307f911245cddcb948512043b5c4f8bb" gracePeriod=30
Jul 28 02:01:13.048518 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:13.048458    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kube-rbac-proxy" containerID="cri-o://ad7525e8bd441d50aa88a8704ef2a5a001f4b7a96ea1a469d5c0a8e544407ad7" gracePeriod=30
Jul 28 02:01:13.448966 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:13.448875    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 02:01:13.946141 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:13.946101    2555 generic.go:356] "Generic (PLEG): container finished" podID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerID="ad7525e8bd441d50aa88a8704ef2a5a001f4b7a96ea1a469d5c0a8e544407ad7" exitCode=2
Jul 28 02:01:13.946326 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:13.946185    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" event={"ID":"8f77be49-28a3-4860-a7a3-766b28efb55f","Type":"ContainerDied","Data":"ad7525e8bd441d50aa88a8704ef2a5a001f4b7a96ea1a469d5c0a8e544407ad7"}
Jul 28 02:01:13.948445 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:13.948422    2555 generic.go:356] "Generic (PLEG): container finished" podID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerID="17760b756e68e9cf4abbf486179b6faefa110359884273a22674eef302c304b6" exitCode=2
Jul 28 02:01:13.948581 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:13.948497    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" event={"ID":"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488","Type":"ContainerDied","Data":"17760b756e68e9cf4abbf486179b6faefa110359884273a22674eef302c304b6"}
Jul 28 02:01:14.660105 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:14.660061    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 28 02:01:14.666330 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:14.666289    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 02:01:16.668278 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:16.668229    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 28 02:01:17.504667 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.504641    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 02:01:17.507413 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.507394    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 02:01:17.578027 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.577961    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz5w5\" (UniqueName: \"kubernetes.io/projected/8f77be49-28a3-4860-a7a3-766b28efb55f-kube-api-access-lz5w5\") pod \"8f77be49-28a3-4860-a7a3-766b28efb55f\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") "
Jul 28 02:01:17.578027 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.577997    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-cabundle-cert\") pod \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") "
Jul 28 02:01:17.578027 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.578025    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kube-rbac-proxy-sar-config\") pod \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") "
Jul 28 02:01:17.578408 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.578046    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq75n\" (UniqueName: \"kubernetes.io/projected/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kube-api-access-cq75n\") pod \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") "
Jul 28 02:01:17.578408 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.578066    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f77be49-28a3-4860-a7a3-766b28efb55f-proxy-tls\") pod \"8f77be49-28a3-4860-a7a3-766b28efb55f\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") "
Jul 28 02:01:17.578408 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.578086    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f77be49-28a3-4860-a7a3-766b28efb55f-kserve-provision-location\") pod \"8f77be49-28a3-4860-a7a3-766b28efb55f\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") "
Jul 28 02:01:17.578408 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.578100    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f77be49-28a3-4860-a7a3-766b28efb55f-cabundle-cert\") pod \"8f77be49-28a3-4860-a7a3-766b28efb55f\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") "
Jul 28 02:01:17.578408 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.578127    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kserve-provision-location\") pod \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") "
Jul 28 02:01:17.578408 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.578176    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-proxy-tls\") pod \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\" (UID: \"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488\") "
Jul 28 02:01:17.578408 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.578212    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f77be49-28a3-4860-a7a3-766b28efb55f-kube-rbac-proxy-sar-config\") pod \"8f77be49-28a3-4860-a7a3-766b28efb55f\" (UID: \"8f77be49-28a3-4860-a7a3-766b28efb55f\") "
Jul 28 02:01:17.578806 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.578437    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" (UID: "de2a26bb-3bf1-4e1b-b1b0-3d332cce2488"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:01:17.578806 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.578471    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" (UID: "de2a26bb-3bf1-4e1b-b1b0-3d332cce2488"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:01:17.578806 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.578710    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f77be49-28a3-4860-a7a3-766b28efb55f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f77be49-28a3-4860-a7a3-766b28efb55f" (UID: "8f77be49-28a3-4860-a7a3-766b28efb55f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:01:17.578971 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.578802    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f77be49-28a3-4860-a7a3-766b28efb55f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f77be49-28a3-4860-a7a3-766b28efb55f" (UID: "8f77be49-28a3-4860-a7a3-766b28efb55f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:01:17.578971 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.578763    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" (UID: "de2a26bb-3bf1-4e1b-b1b0-3d332cce2488"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:01:17.578971 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.578929    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f77be49-28a3-4860-a7a3-766b28efb55f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f77be49-28a3-4860-a7a3-766b28efb55f" (UID: "8f77be49-28a3-4860-a7a3-766b28efb55f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:01:17.580406 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.580374    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kube-api-access-cq75n" (OuterVolumeSpecName: "kube-api-access-cq75n") pod "de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" (UID: "de2a26bb-3bf1-4e1b-b1b0-3d332cce2488"). InnerVolumeSpecName "kube-api-access-cq75n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:01:17.580406 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.580398    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f77be49-28a3-4860-a7a3-766b28efb55f-kube-api-access-lz5w5" (OuterVolumeSpecName: "kube-api-access-lz5w5") pod "8f77be49-28a3-4860-a7a3-766b28efb55f" (UID: "8f77be49-28a3-4860-a7a3-766b28efb55f"). InnerVolumeSpecName "kube-api-access-lz5w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:01:17.580708 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.580683    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f77be49-28a3-4860-a7a3-766b28efb55f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f77be49-28a3-4860-a7a3-766b28efb55f" (UID: "8f77be49-28a3-4860-a7a3-766b28efb55f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:01:17.580781 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.580764    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" (UID: "de2a26bb-3bf1-4e1b-b1b0-3d332cce2488"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:01:17.679235 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.679213    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f77be49-28a3-4860-a7a3-766b28efb55f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:01:17.679235 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.679233    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz5w5\" (UniqueName: \"kubernetes.io/projected/8f77be49-28a3-4860-a7a3-766b28efb55f-kube-api-access-lz5w5\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:01:17.679658 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.679243    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:01:17.679658 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.679252    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:01:17.679658 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.679260    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq75n\" (UniqueName: \"kubernetes.io/projected/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kube-api-access-cq75n\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:01:17.679658 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.679269    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f77be49-28a3-4860-a7a3-766b28efb55f-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:01:17.679658 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.679277    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f77be49-28a3-4860-a7a3-766b28efb55f-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:01:17.679658 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.679284    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f77be49-28a3-4860-a7a3-766b28efb55f-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:01:17.679658 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.679292    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:01:17.679658 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.679299    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:01:17.963237 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.963170    2555 generic.go:356] "Generic (PLEG): container finished" podID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerID="9cff9ddae8297e66d578541bbf4cd654307f911245cddcb948512043b5c4f8bb" exitCode=0
Jul 28 02:01:17.963237 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.963229    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"
Jul 28 02:01:17.963403 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.963255    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" event={"ID":"8f77be49-28a3-4860-a7a3-766b28efb55f","Type":"ContainerDied","Data":"9cff9ddae8297e66d578541bbf4cd654307f911245cddcb948512043b5c4f8bb"}
Jul 28 02:01:17.963403 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.963317    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr" event={"ID":"8f77be49-28a3-4860-a7a3-766b28efb55f","Type":"ContainerDied","Data":"8083d5db2d3bd27138ace0800f7fc3e7f3a429dd7170b98eab6a57b4220c5dc5"}
Jul 28 02:01:17.963403 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.963357    2555 scope.go:117] "RemoveContainer" containerID="ad7525e8bd441d50aa88a8704ef2a5a001f4b7a96ea1a469d5c0a8e544407ad7"
Jul 28 02:01:17.965208 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.965186    2555 generic.go:356] "Generic (PLEG): container finished" podID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerID="da15d3041634333051498a4334b0d83dec224c8e82d14b964ce73db4b0e31f03" exitCode=0
Jul 28 02:01:17.965304 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.965231    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"
Jul 28 02:01:17.965304 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.965242    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" event={"ID":"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488","Type":"ContainerDied","Data":"da15d3041634333051498a4334b0d83dec224c8e82d14b964ce73db4b0e31f03"}
Jul 28 02:01:17.965304 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.965295    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb" event={"ID":"de2a26bb-3bf1-4e1b-b1b0-3d332cce2488","Type":"ContainerDied","Data":"66b51fbcae1b7dbb90093950e8c3132a2d02414fa0b2d5818dd28bb311bbad47"}
Jul 28 02:01:17.971626 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.971604    2555 scope.go:117] "RemoveContainer" containerID="9cff9ddae8297e66d578541bbf4cd654307f911245cddcb948512043b5c4f8bb"
Jul 28 02:01:17.979118 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.979103    2555 scope.go:117] "RemoveContainer" containerID="923f2bc98ef62bc74cddbec3ee045d82269e559acf05f0f4cc46c99fe8d55e8a"
Jul 28 02:01:17.985757 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.985739    2555 scope.go:117] "RemoveContainer" containerID="ad7525e8bd441d50aa88a8704ef2a5a001f4b7a96ea1a469d5c0a8e544407ad7"
Jul 28 02:01:17.985996 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:01:17.985978    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7525e8bd441d50aa88a8704ef2a5a001f4b7a96ea1a469d5c0a8e544407ad7\": container with ID starting with ad7525e8bd441d50aa88a8704ef2a5a001f4b7a96ea1a469d5c0a8e544407ad7 not found: ID does not exist" containerID="ad7525e8bd441d50aa88a8704ef2a5a001f4b7a96ea1a469d5c0a8e544407ad7"
Jul 28 02:01:17.986078 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.986003    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7525e8bd441d50aa88a8704ef2a5a001f4b7a96ea1a469d5c0a8e544407ad7"} err="failed to get container status \"ad7525e8bd441d50aa88a8704ef2a5a001f4b7a96ea1a469d5c0a8e544407ad7\": rpc error: code = NotFound desc = could not find container \"ad7525e8bd441d50aa88a8704ef2a5a001f4b7a96ea1a469d5c0a8e544407ad7\": container with ID starting with ad7525e8bd441d50aa88a8704ef2a5a001f4b7a96ea1a469d5c0a8e544407ad7 not found: ID does not exist"
Jul 28 02:01:17.986078 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.986019    2555 scope.go:117] "RemoveContainer" containerID="9cff9ddae8297e66d578541bbf4cd654307f911245cddcb948512043b5c4f8bb"
Jul 28 02:01:17.986236 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:01:17.986217    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cff9ddae8297e66d578541bbf4cd654307f911245cddcb948512043b5c4f8bb\": container with ID starting with 9cff9ddae8297e66d578541bbf4cd654307f911245cddcb948512043b5c4f8bb not found: ID does not exist" containerID="9cff9ddae8297e66d578541bbf4cd654307f911245cddcb948512043b5c4f8bb"
Jul 28 02:01:17.986297 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.986241    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cff9ddae8297e66d578541bbf4cd654307f911245cddcb948512043b5c4f8bb"} err="failed to get container status \"9cff9ddae8297e66d578541bbf4cd654307f911245cddcb948512043b5c4f8bb\": rpc error: code = NotFound desc = could not find container \"9cff9ddae8297e66d578541bbf4cd654307f911245cddcb948512043b5c4f8bb\": container with ID starting with 9cff9ddae8297e66d578541bbf4cd654307f911245cddcb948512043b5c4f8bb not found: ID does not exist"
Jul 28 02:01:17.986297 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.986253    2555 scope.go:117] "RemoveContainer" containerID="923f2bc98ef62bc74cddbec3ee045d82269e559acf05f0f4cc46c99fe8d55e8a"
Jul 28 02:01:17.986440 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:01:17.986421    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"923f2bc98ef62bc74cddbec3ee045d82269e559acf05f0f4cc46c99fe8d55e8a\": container with ID starting with 923f2bc98ef62bc74cddbec3ee045d82269e559acf05f0f4cc46c99fe8d55e8a not found: ID does not exist" containerID="923f2bc98ef62bc74cddbec3ee045d82269e559acf05f0f4cc46c99fe8d55e8a"
Jul 28 02:01:17.986491 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.986443    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"923f2bc98ef62bc74cddbec3ee045d82269e559acf05f0f4cc46c99fe8d55e8a"} err="failed to get container status \"923f2bc98ef62bc74cddbec3ee045d82269e559acf05f0f4cc46c99fe8d55e8a\": rpc error: code = NotFound desc = could not find container \"923f2bc98ef62bc74cddbec3ee045d82269e559acf05f0f4cc46c99fe8d55e8a\": container with ID starting with 923f2bc98ef62bc74cddbec3ee045d82269e559acf05f0f4cc46c99fe8d55e8a not found: ID does not exist"
Jul 28 02:01:17.986491 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.986455    2555 scope.go:117] "RemoveContainer" containerID="17760b756e68e9cf4abbf486179b6faefa110359884273a22674eef302c304b6"
Jul 28 02:01:17.990457 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.990439    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"]
Jul 28 02:01:17.994461 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.994445    2555 scope.go:117] "RemoveContainer" containerID="da15d3041634333051498a4334b0d83dec224c8e82d14b964ce73db4b0e31f03"
Jul 28 02:01:17.995217 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:17.995199    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-f92zr"]
Jul 28 02:01:18.001509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:18.001489    2555 scope.go:117] "RemoveContainer" containerID="5ebda43edcc2001b4e54b0dcfd967947f05d3a329490fdde28a7bea1528d4b3d"
Jul 28 02:01:18.005883 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:18.005864    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"]
Jul 28 02:01:18.008834 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:18.008820    2555 scope.go:117] "RemoveContainer" containerID="17760b756e68e9cf4abbf486179b6faefa110359884273a22674eef302c304b6"
Jul 28 02:01:18.009069 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:01:18.009047    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17760b756e68e9cf4abbf486179b6faefa110359884273a22674eef302c304b6\": container with ID starting with 17760b756e68e9cf4abbf486179b6faefa110359884273a22674eef302c304b6 not found: ID does not exist" containerID="17760b756e68e9cf4abbf486179b6faefa110359884273a22674eef302c304b6"
Jul 28 02:01:18.009120 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:18.009075    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17760b756e68e9cf4abbf486179b6faefa110359884273a22674eef302c304b6"} err="failed to get container status \"17760b756e68e9cf4abbf486179b6faefa110359884273a22674eef302c304b6\": rpc error: code = NotFound desc = could not find container \"17760b756e68e9cf4abbf486179b6faefa110359884273a22674eef302c304b6\": container with ID starting with 17760b756e68e9cf4abbf486179b6faefa110359884273a22674eef302c304b6 not found: ID does not exist"
Jul 28 02:01:18.009120 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:18.009090    2555 scope.go:117] "RemoveContainer" containerID="da15d3041634333051498a4334b0d83dec224c8e82d14b964ce73db4b0e31f03"
Jul 28 02:01:18.009321 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:01:18.009305    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da15d3041634333051498a4334b0d83dec224c8e82d14b964ce73db4b0e31f03\": container with ID starting with da15d3041634333051498a4334b0d83dec224c8e82d14b964ce73db4b0e31f03 not found: ID does not exist" containerID="da15d3041634333051498a4334b0d83dec224c8e82d14b964ce73db4b0e31f03"
Jul 28 02:01:18.009361 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:18.009327    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da15d3041634333051498a4334b0d83dec224c8e82d14b964ce73db4b0e31f03"} err="failed to get container status \"da15d3041634333051498a4334b0d83dec224c8e82d14b964ce73db4b0e31f03\": rpc error: code = NotFound desc = could not find container \"da15d3041634333051498a4334b0d83dec224c8e82d14b964ce73db4b0e31f03\": container with ID starting with da15d3041634333051498a4334b0d83dec224c8e82d14b964ce73db4b0e31f03 not found: ID does not exist"
Jul 28 02:01:18.009361 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:18.009341    2555 scope.go:117] "RemoveContainer" containerID="5ebda43edcc2001b4e54b0dcfd967947f05d3a329490fdde28a7bea1528d4b3d"
Jul 28 02:01:18.009676 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:01:18.009655    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ebda43edcc2001b4e54b0dcfd967947f05d3a329490fdde28a7bea1528d4b3d\": container with ID starting with 5ebda43edcc2001b4e54b0dcfd967947f05d3a329490fdde28a7bea1528d4b3d not found: ID does not exist" containerID="5ebda43edcc2001b4e54b0dcfd967947f05d3a329490fdde28a7bea1528d4b3d"
Jul 28 02:01:18.009771 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:18.009685    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebda43edcc2001b4e54b0dcfd967947f05d3a329490fdde28a7bea1528d4b3d"} err="failed to get container status \"5ebda43edcc2001b4e54b0dcfd967947f05d3a329490fdde28a7bea1528d4b3d\": rpc error: code = NotFound desc = could not find container \"5ebda43edcc2001b4e54b0dcfd967947f05d3a329490fdde28a7bea1528d4b3d\": container with ID starting with 5ebda43edcc2001b4e54b0dcfd967947f05d3a329490fdde28a7bea1528d4b3d not found: ID does not exist"
Jul 28 02:01:18.011443 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:18.011422    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-csttb"]
Jul 28 02:01:19.451362 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:19.451328    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" path="/var/lib/kubelet/pods/8f77be49-28a3-4860-a7a3-766b28efb55f/volumes"
Jul 28 02:01:19.451809 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:19.451795    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" path="/var/lib/kubelet/pods/de2a26bb-3bf1-4e1b-b1b0-3d332cce2488/volumes"
Jul 28 02:01:23.324060 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324023    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"]
Jul 28 02:01:23.324412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324253    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kube-rbac-proxy"
Jul 28 02:01:23.324412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324263    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kube-rbac-proxy"
Jul 28 02:01:23.324412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324275    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="storage-initializer"
Jul 28 02:01:23.324412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324280    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="storage-initializer"
Jul 28 02:01:23.324412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324289    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kube-rbac-proxy"
Jul 28 02:01:23.324412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324295    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kube-rbac-proxy"
Jul 28 02:01:23.324412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324301    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kserve-container"
Jul 28 02:01:23.324412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324306    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kserve-container"
Jul 28 02:01:23.324412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324313    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="storage-initializer"
Jul 28 02:01:23.324412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324318    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="storage-initializer"
Jul 28 02:01:23.324412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324325    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kserve-container"
Jul 28 02:01:23.324412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324330    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kserve-container"
Jul 28 02:01:23.324412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324372    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kube-rbac-proxy"
Jul 28 02:01:23.324412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324379    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kserve-container"
Jul 28 02:01:23.324412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324385    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f77be49-28a3-4860-a7a3-766b28efb55f" containerName="kube-rbac-proxy"
Jul 28 02:01:23.324412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.324390    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2a26bb-3bf1-4e1b-b1b0-3d332cce2488" containerName="kserve-container"
Jul 28 02:01:23.328982 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.328963    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.331374 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.331352    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jul 28 02:01:23.331660 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.331638    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:01:23.331792 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.331690    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jul 28 02:01:23.331792 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.331700    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:01:23.331792 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.331701    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:01:23.331792 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.331704    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:01:23.332690 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.332675    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:01:23.337920 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.337895    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"]
Jul 28 02:01:23.417995 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.417960    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ec436c-8240-4793-afe7-e2e66b43b22b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-mddkn\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.418123 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.418009    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ec436c-8240-4793-afe7-e2e66b43b22b-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-mddkn\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.418123 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.418054    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pdvz\" (UniqueName: \"kubernetes.io/projected/f5ec436c-8240-4793-afe7-e2e66b43b22b-kube-api-access-6pdvz\") pod \"isvc-lightgbm-predictor-54f85ddf88-mddkn\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.418123 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.418108    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ec436c-8240-4793-afe7-e2e66b43b22b-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-mddkn\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.418247 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.418133    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ec436c-8240-4793-afe7-e2e66b43b22b-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-mddkn\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.518730 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.518704    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ec436c-8240-4793-afe7-e2e66b43b22b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-mddkn\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.518850 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.518738    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ec436c-8240-4793-afe7-e2e66b43b22b-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-mddkn\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.518850 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.518757    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pdvz\" (UniqueName: \"kubernetes.io/projected/f5ec436c-8240-4793-afe7-e2e66b43b22b-kube-api-access-6pdvz\") pod \"isvc-lightgbm-predictor-54f85ddf88-mddkn\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.518850 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.518787    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ec436c-8240-4793-afe7-e2e66b43b22b-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-mddkn\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.519022 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.518905    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ec436c-8240-4793-afe7-e2e66b43b22b-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-mddkn\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.519194 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.519177    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ec436c-8240-4793-afe7-e2e66b43b22b-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-mddkn\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.519517 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.519492    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ec436c-8240-4793-afe7-e2e66b43b22b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-mddkn\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.519517 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.519510    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ec436c-8240-4793-afe7-e2e66b43b22b-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-mddkn\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.521394 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.521371    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ec436c-8240-4793-afe7-e2e66b43b22b-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-mddkn\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.527696 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.527672    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pdvz\" (UniqueName: \"kubernetes.io/projected/f5ec436c-8240-4793-afe7-e2e66b43b22b-kube-api-access-6pdvz\") pod \"isvc-lightgbm-predictor-54f85ddf88-mddkn\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.639910 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.639844    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:23.769778 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.769583    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"]
Jul 28 02:01:23.772305 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:01:23.772271    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ec436c_8240_4793_afe7_e2e66b43b22b.slice/crio-0d30851c3fd83e400c82dbcf95cfd2fce599f040814ad797d3a1a72487e26082 WatchSource:0}: Error finding container 0d30851c3fd83e400c82dbcf95cfd2fce599f040814ad797d3a1a72487e26082: Status 404 returned error can't find the container with id 0d30851c3fd83e400c82dbcf95cfd2fce599f040814ad797d3a1a72487e26082
Jul 28 02:01:23.989898 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.989861    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" event={"ID":"f5ec436c-8240-4793-afe7-e2e66b43b22b","Type":"ContainerStarted","Data":"857a9e819e33dc745dfdd33ade22c855a1994611d58502255610932287279873"}
Jul 28 02:01:23.990051 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:23.989937    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" event={"ID":"f5ec436c-8240-4793-afe7-e2e66b43b22b","Type":"ContainerStarted","Data":"0d30851c3fd83e400c82dbcf95cfd2fce599f040814ad797d3a1a72487e26082"}
Jul 28 02:01:28.003905 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:28.003875    2555 generic.go:356] "Generic (PLEG): container finished" podID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerID="857a9e819e33dc745dfdd33ade22c855a1994611d58502255610932287279873" exitCode=0
Jul 28 02:01:28.004250 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:28.003937    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" event={"ID":"f5ec436c-8240-4793-afe7-e2e66b43b22b","Type":"ContainerDied","Data":"857a9e819e33dc745dfdd33ade22c855a1994611d58502255610932287279873"}
Jul 28 02:01:28.005187 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:28.005170    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 02:01:35.040968 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:35.040928    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" event={"ID":"f5ec436c-8240-4793-afe7-e2e66b43b22b","Type":"ContainerStarted","Data":"d2723151dbd2e25a2bc4d7daa7dcc4f4135c920c25e7d5f067f56d1a7330e901"}
Jul 28 02:01:35.041363 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:35.041014    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" event={"ID":"f5ec436c-8240-4793-afe7-e2e66b43b22b","Type":"ContainerStarted","Data":"d0883812b3aab7928d47f000b311f1173c5539ef00da0eb87c8112aeb5ca0e3e"}
Jul 28 02:01:35.041363 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:35.041276    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:35.041473 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:35.041415    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:35.042344 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:35.042317    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 02:01:35.061287 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:35.061180    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" podStartSLOduration=2.003077214 podStartE2EDuration="12.061163904s" totalImagesPullingTime="6.259985068s" totalInitContainerRuntime="3.798101622s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:01:35.060924365 +0000 UTC m=+2142.070442816" watchObservedRunningTime="2026-07-28 02:01:35.061163904 +0000 UTC m=+2142.070682316"
Jul 28 02:01:36.044482 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:36.044446    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 02:01:41.048891 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:41.048863    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:01:41.049350 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:41.049323    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 02:01:51.049627 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:01:51.049516    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 02:02:01.050219 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:01.050179    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 02:02:11.050286 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:11.050247    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 02:02:21.049822 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:21.049783    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 02:02:31.049811 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:31.049772    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 02:02:41.049727 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:41.049688    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 02:02:51.050722 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:51.050688    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:02:53.523576 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:53.523543    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"]
Jul 28 02:02:53.523985 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:53.523789    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kserve-container" containerID="cri-o://d0883812b3aab7928d47f000b311f1173c5539ef00da0eb87c8112aeb5ca0e3e" gracePeriod=30
Jul 28 02:02:53.523985 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:53.523850    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kube-rbac-proxy" containerID="cri-o://d2723151dbd2e25a2bc4d7daa7dcc4f4135c920c25e7d5f067f56d1a7330e901" gracePeriod=30
Jul 28 02:02:54.312465 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:54.312428    2555 generic.go:356] "Generic (PLEG): container finished" podID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerID="d2723151dbd2e25a2bc4d7daa7dcc4f4135c920c25e7d5f067f56d1a7330e901" exitCode=2
Jul 28 02:02:54.312649 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:54.312501    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" event={"ID":"f5ec436c-8240-4793-afe7-e2e66b43b22b","Type":"ContainerDied","Data":"d2723151dbd2e25a2bc4d7daa7dcc4f4135c920c25e7d5f067f56d1a7330e901"}
Jul 28 02:02:56.045016 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:56.044973    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 28 02:02:58.266721 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.266696    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:02:58.272176 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.272157    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ec436c-8240-4793-afe7-e2e66b43b22b-kube-rbac-proxy-sar-config\") pod \"f5ec436c-8240-4793-afe7-e2e66b43b22b\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") "
Jul 28 02:02:58.272288 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.272217    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ec436c-8240-4793-afe7-e2e66b43b22b-kserve-provision-location\") pod \"f5ec436c-8240-4793-afe7-e2e66b43b22b\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") "
Jul 28 02:02:58.272288 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.272258    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ec436c-8240-4793-afe7-e2e66b43b22b-proxy-tls\") pod \"f5ec436c-8240-4793-afe7-e2e66b43b22b\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") "
Jul 28 02:02:58.272400 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.272288    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pdvz\" (UniqueName: \"kubernetes.io/projected/f5ec436c-8240-4793-afe7-e2e66b43b22b-kube-api-access-6pdvz\") pod \"f5ec436c-8240-4793-afe7-e2e66b43b22b\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") "
Jul 28 02:02:58.272400 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.272335    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ec436c-8240-4793-afe7-e2e66b43b22b-cabundle-cert\") pod \"f5ec436c-8240-4793-afe7-e2e66b43b22b\" (UID: \"f5ec436c-8240-4793-afe7-e2e66b43b22b\") "
Jul 28 02:02:58.272578 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.272553    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ec436c-8240-4793-afe7-e2e66b43b22b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f5ec436c-8240-4793-afe7-e2e66b43b22b" (UID: "f5ec436c-8240-4793-afe7-e2e66b43b22b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:02:58.272658 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.272602    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ec436c-8240-4793-afe7-e2e66b43b22b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5ec436c-8240-4793-afe7-e2e66b43b22b" (UID: "f5ec436c-8240-4793-afe7-e2e66b43b22b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:02:58.272752 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.272731    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ec436c-8240-4793-afe7-e2e66b43b22b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5ec436c-8240-4793-afe7-e2e66b43b22b" (UID: "f5ec436c-8240-4793-afe7-e2e66b43b22b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:02:58.274465 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.274438    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ec436c-8240-4793-afe7-e2e66b43b22b-kube-api-access-6pdvz" (OuterVolumeSpecName: "kube-api-access-6pdvz") pod "f5ec436c-8240-4793-afe7-e2e66b43b22b" (UID: "f5ec436c-8240-4793-afe7-e2e66b43b22b"). InnerVolumeSpecName "kube-api-access-6pdvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:02:58.274544 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.274501    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ec436c-8240-4793-afe7-e2e66b43b22b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5ec436c-8240-4793-afe7-e2e66b43b22b" (UID: "f5ec436c-8240-4793-afe7-e2e66b43b22b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:02:58.327823 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.327759    2555 generic.go:356] "Generic (PLEG): container finished" podID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerID="d0883812b3aab7928d47f000b311f1173c5539ef00da0eb87c8112aeb5ca0e3e" exitCode=0
Jul 28 02:02:58.327823 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.327818    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"
Jul 28 02:02:58.327969 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.327837    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" event={"ID":"f5ec436c-8240-4793-afe7-e2e66b43b22b","Type":"ContainerDied","Data":"d0883812b3aab7928d47f000b311f1173c5539ef00da0eb87c8112aeb5ca0e3e"}
Jul 28 02:02:58.327969 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.327915    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn" event={"ID":"f5ec436c-8240-4793-afe7-e2e66b43b22b","Type":"ContainerDied","Data":"0d30851c3fd83e400c82dbcf95cfd2fce599f040814ad797d3a1a72487e26082"}
Jul 28 02:02:58.327969 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.327951    2555 scope.go:117] "RemoveContainer" containerID="d2723151dbd2e25a2bc4d7daa7dcc4f4135c920c25e7d5f067f56d1a7330e901"
Jul 28 02:02:58.337376 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.337350    2555 scope.go:117] "RemoveContainer" containerID="d0883812b3aab7928d47f000b311f1173c5539ef00da0eb87c8112aeb5ca0e3e"
Jul 28 02:02:58.344479 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.344462    2555 scope.go:117] "RemoveContainer" containerID="857a9e819e33dc745dfdd33ade22c855a1994611d58502255610932287279873"
Jul 28 02:02:58.351569 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.351551    2555 scope.go:117] "RemoveContainer" containerID="d2723151dbd2e25a2bc4d7daa7dcc4f4135c920c25e7d5f067f56d1a7330e901"
Jul 28 02:02:58.351873 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:02:58.351844    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2723151dbd2e25a2bc4d7daa7dcc4f4135c920c25e7d5f067f56d1a7330e901\": container with ID starting with d2723151dbd2e25a2bc4d7daa7dcc4f4135c920c25e7d5f067f56d1a7330e901 not found: ID does not exist" containerID="d2723151dbd2e25a2bc4d7daa7dcc4f4135c920c25e7d5f067f56d1a7330e901"
Jul 28 02:02:58.351957 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.351873    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2723151dbd2e25a2bc4d7daa7dcc4f4135c920c25e7d5f067f56d1a7330e901"} err="failed to get container status \"d2723151dbd2e25a2bc4d7daa7dcc4f4135c920c25e7d5f067f56d1a7330e901\": rpc error: code = NotFound desc = could not find container \"d2723151dbd2e25a2bc4d7daa7dcc4f4135c920c25e7d5f067f56d1a7330e901\": container with ID starting with d2723151dbd2e25a2bc4d7daa7dcc4f4135c920c25e7d5f067f56d1a7330e901 not found: ID does not exist"
Jul 28 02:02:58.351957 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.351890    2555 scope.go:117] "RemoveContainer" containerID="d0883812b3aab7928d47f000b311f1173c5539ef00da0eb87c8112aeb5ca0e3e"
Jul 28 02:02:58.352144 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:02:58.352126    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0883812b3aab7928d47f000b311f1173c5539ef00da0eb87c8112aeb5ca0e3e\": container with ID starting with d0883812b3aab7928d47f000b311f1173c5539ef00da0eb87c8112aeb5ca0e3e not found: ID does not exist" containerID="d0883812b3aab7928d47f000b311f1173c5539ef00da0eb87c8112aeb5ca0e3e"
Jul 28 02:02:58.352190 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.352150    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0883812b3aab7928d47f000b311f1173c5539ef00da0eb87c8112aeb5ca0e3e"} err="failed to get container status \"d0883812b3aab7928d47f000b311f1173c5539ef00da0eb87c8112aeb5ca0e3e\": rpc error: code = NotFound desc = could not find container \"d0883812b3aab7928d47f000b311f1173c5539ef00da0eb87c8112aeb5ca0e3e\": container with ID starting with d0883812b3aab7928d47f000b311f1173c5539ef00da0eb87c8112aeb5ca0e3e not found: ID does not exist"
Jul 28 02:02:58.352190 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.352166    2555 scope.go:117] "RemoveContainer" containerID="857a9e819e33dc745dfdd33ade22c855a1994611d58502255610932287279873"
Jul 28 02:02:58.352415 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:02:58.352395    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857a9e819e33dc745dfdd33ade22c855a1994611d58502255610932287279873\": container with ID starting with 857a9e819e33dc745dfdd33ade22c855a1994611d58502255610932287279873 not found: ID does not exist" containerID="857a9e819e33dc745dfdd33ade22c855a1994611d58502255610932287279873"
Jul 28 02:02:58.352457 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.352425    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857a9e819e33dc745dfdd33ade22c855a1994611d58502255610932287279873"} err="failed to get container status \"857a9e819e33dc745dfdd33ade22c855a1994611d58502255610932287279873\": rpc error: code = NotFound desc = could not find container \"857a9e819e33dc745dfdd33ade22c855a1994611d58502255610932287279873\": container with ID starting with 857a9e819e33dc745dfdd33ade22c855a1994611d58502255610932287279873 not found: ID does not exist"
Jul 28 02:02:58.354964 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.354943    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"]
Jul 28 02:02:58.357446 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.357425    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mddkn"]
Jul 28 02:02:58.373575 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.373558    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ec436c-8240-4793-afe7-e2e66b43b22b-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:02:58.373654 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.373577    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ec436c-8240-4793-afe7-e2e66b43b22b-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:02:58.373654 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.373589    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pdvz\" (UniqueName: \"kubernetes.io/projected/f5ec436c-8240-4793-afe7-e2e66b43b22b-kube-api-access-6pdvz\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:02:58.373654 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.373598    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ec436c-8240-4793-afe7-e2e66b43b22b-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:02:58.373654 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.373607    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ec436c-8240-4793-afe7-e2e66b43b22b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:02:58.715329 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.715250    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"]
Jul 28 02:02:58.715511 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.715499    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kserve-container"
Jul 28 02:02:58.715582 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.715513    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kserve-container"
Jul 28 02:02:58.715582 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.715558    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kube-rbac-proxy"
Jul 28 02:02:58.715582 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.715567    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kube-rbac-proxy"
Jul 28 02:02:58.715582 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.715576    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="storage-initializer"
Jul 28 02:02:58.715582 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.715583    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="storage-initializer"
Jul 28 02:02:58.715735 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.715626    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kube-rbac-proxy"
Jul 28 02:02:58.715735 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.715635    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" containerName="kserve-container"
Jul 28 02:02:58.718512 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.718495    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:58.721416 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.721392    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:02:58.721545 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.721417    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:02:58.721545 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.721416    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:02:58.721793 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.721776    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jul 28 02:02:58.721853 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.721784    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:02:58.721853 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.721818    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:02:58.722038 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.722022    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jul 28 02:02:58.736938 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.736916    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"]
Jul 28 02:02:58.775837 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.775812    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:58.775951 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.775852    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdgd9\" (UniqueName: \"kubernetes.io/projected/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kube-api-access-wdgd9\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:58.775951 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.775875    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:58.776025 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.775951    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:58.776025 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.775986    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:58.876874 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.876851    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:58.876987 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.876882    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:58.876987 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.876903    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdgd9\" (UniqueName: \"kubernetes.io/projected/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kube-api-access-wdgd9\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:58.877061 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:02:58.876996    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Jul 28 02:02:58.877061 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:02:58.877053    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-proxy-tls podName:ae6e4975-e39f-411c-bab4-22f5dbd8ebc2 nodeName:}" failed. No retries permitted until 2026-07-28 02:02:59.377035656 +0000 UTC m=+2226.386554064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" (UID: "ae6e4975-e39f-411c-bab4-22f5dbd8ebc2") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Jul 28 02:02:58.877241 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.877217    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:58.877339 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.877298    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:58.877611 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.877586    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:58.877707 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.877650    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:58.877907 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.877887    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:58.892669 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:58.892651    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdgd9\" (UniqueName: \"kubernetes.io/projected/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kube-api-access-wdgd9\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:59.379886 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:59.379859    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:59.382344 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:59.382325    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:59.451063 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:59.451032    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5ec436c-8240-4793-afe7-e2e66b43b22b" path="/var/lib/kubelet/pods/f5ec436c-8240-4793-afe7-e2e66b43b22b/volumes"
Jul 28 02:02:59.628411 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:59.628380    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:02:59.762330 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:02:59.762304    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"]
Jul 28 02:02:59.764574 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:02:59.764547    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6e4975_e39f_411c_bab4_22f5dbd8ebc2.slice/crio-c62d45430aaf7206fd3f5c04191b1bcdb14a46e78f699aeaa5357ebade154086 WatchSource:0}: Error finding container c62d45430aaf7206fd3f5c04191b1bcdb14a46e78f699aeaa5357ebade154086: Status 404 returned error can't find the container with id c62d45430aaf7206fd3f5c04191b1bcdb14a46e78f699aeaa5357ebade154086
Jul 28 02:03:00.335864 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:00.335829    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" event={"ID":"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2","Type":"ContainerStarted","Data":"3b9d180dafff412efbbd9e555abdc12f7970dc0bb7fcf3de024ba0432711d946"}
Jul 28 02:03:00.336030 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:00.335887    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" event={"ID":"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2","Type":"ContainerStarted","Data":"c62d45430aaf7206fd3f5c04191b1bcdb14a46e78f699aeaa5357ebade154086"}
Jul 28 02:03:03.348033 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:03.347996    2555 generic.go:356] "Generic (PLEG): container finished" podID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerID="3b9d180dafff412efbbd9e555abdc12f7970dc0bb7fcf3de024ba0432711d946" exitCode=0
Jul 28 02:03:03.348408 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:03.348071    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" event={"ID":"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2","Type":"ContainerDied","Data":"3b9d180dafff412efbbd9e555abdc12f7970dc0bb7fcf3de024ba0432711d946"}
Jul 28 02:03:04.353750 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:04.353710    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" event={"ID":"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2","Type":"ContainerStarted","Data":"7db9cd13ef3620a464c8c9f46dd54bd5ae905b801b9b90fe0a2a2edb204e4446"}
Jul 28 02:03:04.354230 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:04.353782    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" event={"ID":"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2","Type":"ContainerStarted","Data":"c50c0f76bf628e77fa1f003f2eab551c475680462960046b0f87d146686cffc4"}
Jul 28 02:03:04.354230 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:04.354015    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:03:04.375710 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:04.375655    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" podStartSLOduration=2.925401818 podStartE2EDuration="6.375638696s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.450236878s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:03:04.374794385 +0000 UTC m=+2231.384312828" watchObservedRunningTime="2026-07-28 02:03:04.375638696 +0000 UTC m=+2231.385157112"
Jul 28 02:03:05.357904 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:05.357874    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:03:05.359057 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:05.359028    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 02:03:06.361729 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:06.361691    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 02:03:11.366031 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:11.366004    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:03:11.366570 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:11.366541    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 02:03:21.366824 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:21.366740    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 02:03:31.366763 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:31.366719    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 02:03:41.366945 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:41.366907    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 02:03:51.366493 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:03:51.366436    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 02:04:01.366569 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:01.366494    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 02:04:11.367149 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:11.367106    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 02:04:21.367686 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:21.367648    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:04:29.133689 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:29.133651    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"]
Jul 28 02:04:29.134194 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:29.133950    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kserve-container" containerID="cri-o://c50c0f76bf628e77fa1f003f2eab551c475680462960046b0f87d146686cffc4" gracePeriod=30
Jul 28 02:04:29.134194 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:29.134015    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kube-rbac-proxy" containerID="cri-o://7db9cd13ef3620a464c8c9f46dd54bd5ae905b801b9b90fe0a2a2edb204e4446" gracePeriod=30
Jul 28 02:04:29.638722 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:29.638690    2555 generic.go:356] "Generic (PLEG): container finished" podID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerID="7db9cd13ef3620a464c8c9f46dd54bd5ae905b801b9b90fe0a2a2edb204e4446" exitCode=2
Jul 28 02:04:29.638903 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:29.638744    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" event={"ID":"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2","Type":"ContainerDied","Data":"7db9cd13ef3620a464c8c9f46dd54bd5ae905b801b9b90fe0a2a2edb204e4446"}
Jul 28 02:04:31.362389 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:31.362345    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 28 02:04:31.366765 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:31.366735    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 02:04:34.288663 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.288625    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"]
Jul 28 02:04:34.291954 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.291933    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:34.295403 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.295382    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 28 02:04:34.295544 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.295414    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 02:04:34.303762 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.303732    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"]
Jul 28 02:04:34.452276 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.452236    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:34.452454 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.452294    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:34.452454 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.452322    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:34.452454 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.452364    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prvzt\" (UniqueName: \"kubernetes.io/projected/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kube-api-access-prvzt\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:34.452454 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.452393    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:34.472741 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.472718    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:04:34.553518 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.553419    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:34.553518 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.553461    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:34.553518 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.553481    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:34.553898 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.553543    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prvzt\" (UniqueName: \"kubernetes.io/projected/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kube-api-access-prvzt\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:34.553898 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.553573    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:34.553898 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:04:34.553613    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jul 28 02:04:34.553898 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:04:34.553678    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-proxy-tls podName:8732fd8c-41c4-4ad8-8f70-dcd4660b3b49 nodeName:}" failed. No retries permitted until 2026-07-28 02:04:35.053657287 +0000 UTC m=+2322.063175693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f" (UID: "8732fd8c-41c4-4ad8-8f70-dcd4660b3b49") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jul 28 02:04:34.554125 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.553976    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:34.554248 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.554228    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:34.554331 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.554310    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:34.563446 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.563419    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prvzt\" (UniqueName: \"kubernetes.io/projected/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kube-api-access-prvzt\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:34.653968 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.653927    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kserve-provision-location\") pod \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") "
Jul 28 02:04:34.654158 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.653978    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-proxy-tls\") pod \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") "
Jul 28 02:04:34.654158 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.654008    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdgd9\" (UniqueName: \"kubernetes.io/projected/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kube-api-access-wdgd9\") pod \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") "
Jul 28 02:04:34.654158 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.654087    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-cabundle-cert\") pod \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") "
Jul 28 02:04:34.654158 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.654115    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kube-rbac-proxy-sar-config\") pod \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\" (UID: \"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2\") "
Jul 28 02:04:34.654392 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.654272    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" (UID: "ae6e4975-e39f-411c-bab4-22f5dbd8ebc2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:04:34.654491 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.654413    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" (UID: "ae6e4975-e39f-411c-bab4-22f5dbd8ebc2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:04:34.654630 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.654598    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" (UID: "ae6e4975-e39f-411c-bab4-22f5dbd8ebc2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:04:34.656898 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.656866    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" (UID: "ae6e4975-e39f-411c-bab4-22f5dbd8ebc2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:04:34.656898 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.656891    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kube-api-access-wdgd9" (OuterVolumeSpecName: "kube-api-access-wdgd9") pod "ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" (UID: "ae6e4975-e39f-411c-bab4-22f5dbd8ebc2"). InnerVolumeSpecName "kube-api-access-wdgd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:04:34.658658 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.658634    2555 generic.go:356] "Generic (PLEG): container finished" podID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerID="c50c0f76bf628e77fa1f003f2eab551c475680462960046b0f87d146686cffc4" exitCode=0
Jul 28 02:04:34.658747 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.658671    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" event={"ID":"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2","Type":"ContainerDied","Data":"c50c0f76bf628e77fa1f003f2eab551c475680462960046b0f87d146686cffc4"}
Jul 28 02:04:34.658747 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.658714    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"
Jul 28 02:04:34.658747 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.658728    2555 scope.go:117] "RemoveContainer" containerID="7db9cd13ef3620a464c8c9f46dd54bd5ae905b801b9b90fe0a2a2edb204e4446"
Jul 28 02:04:34.658853 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.658719    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9" event={"ID":"ae6e4975-e39f-411c-bab4-22f5dbd8ebc2","Type":"ContainerDied","Data":"c62d45430aaf7206fd3f5c04191b1bcdb14a46e78f699aeaa5357ebade154086"}
Jul 28 02:04:34.667786 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.667590    2555 scope.go:117] "RemoveContainer" containerID="c50c0f76bf628e77fa1f003f2eab551c475680462960046b0f87d146686cffc4"
Jul 28 02:04:34.674616 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.674601    2555 scope.go:117] "RemoveContainer" containerID="3b9d180dafff412efbbd9e555abdc12f7970dc0bb7fcf3de024ba0432711d946"
Jul 28 02:04:34.681496 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.681475    2555 scope.go:117] "RemoveContainer" containerID="7db9cd13ef3620a464c8c9f46dd54bd5ae905b801b9b90fe0a2a2edb204e4446"
Jul 28 02:04:34.681794 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:04:34.681776    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db9cd13ef3620a464c8c9f46dd54bd5ae905b801b9b90fe0a2a2edb204e4446\": container with ID starting with 7db9cd13ef3620a464c8c9f46dd54bd5ae905b801b9b90fe0a2a2edb204e4446 not found: ID does not exist" containerID="7db9cd13ef3620a464c8c9f46dd54bd5ae905b801b9b90fe0a2a2edb204e4446"
Jul 28 02:04:34.681884 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.681806    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db9cd13ef3620a464c8c9f46dd54bd5ae905b801b9b90fe0a2a2edb204e4446"} err="failed to get container status \"7db9cd13ef3620a464c8c9f46dd54bd5ae905b801b9b90fe0a2a2edb204e4446\": rpc error: code = NotFound desc = could not find container \"7db9cd13ef3620a464c8c9f46dd54bd5ae905b801b9b90fe0a2a2edb204e4446\": container with ID starting with 7db9cd13ef3620a464c8c9f46dd54bd5ae905b801b9b90fe0a2a2edb204e4446 not found: ID does not exist"
Jul 28 02:04:34.681884 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.681830    2555 scope.go:117] "RemoveContainer" containerID="c50c0f76bf628e77fa1f003f2eab551c475680462960046b0f87d146686cffc4"
Jul 28 02:04:34.682052 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:04:34.682036    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50c0f76bf628e77fa1f003f2eab551c475680462960046b0f87d146686cffc4\": container with ID starting with c50c0f76bf628e77fa1f003f2eab551c475680462960046b0f87d146686cffc4 not found: ID does not exist" containerID="c50c0f76bf628e77fa1f003f2eab551c475680462960046b0f87d146686cffc4"
Jul 28 02:04:34.682100 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.682058    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50c0f76bf628e77fa1f003f2eab551c475680462960046b0f87d146686cffc4"} err="failed to get container status \"c50c0f76bf628e77fa1f003f2eab551c475680462960046b0f87d146686cffc4\": rpc error: code = NotFound desc = could not find container \"c50c0f76bf628e77fa1f003f2eab551c475680462960046b0f87d146686cffc4\": container with ID starting with c50c0f76bf628e77fa1f003f2eab551c475680462960046b0f87d146686cffc4 not found: ID does not exist"
Jul 28 02:04:34.682100 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.682074    2555 scope.go:117] "RemoveContainer" containerID="3b9d180dafff412efbbd9e555abdc12f7970dc0bb7fcf3de024ba0432711d946"
Jul 28 02:04:34.682265 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:04:34.682247    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9d180dafff412efbbd9e555abdc12f7970dc0bb7fcf3de024ba0432711d946\": container with ID starting with 3b9d180dafff412efbbd9e555abdc12f7970dc0bb7fcf3de024ba0432711d946 not found: ID does not exist" containerID="3b9d180dafff412efbbd9e555abdc12f7970dc0bb7fcf3de024ba0432711d946"
Jul 28 02:04:34.682318 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.682271    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9d180dafff412efbbd9e555abdc12f7970dc0bb7fcf3de024ba0432711d946"} err="failed to get container status \"3b9d180dafff412efbbd9e555abdc12f7970dc0bb7fcf3de024ba0432711d946\": rpc error: code = NotFound desc = could not find container \"3b9d180dafff412efbbd9e555abdc12f7970dc0bb7fcf3de024ba0432711d946\": container with ID starting with 3b9d180dafff412efbbd9e555abdc12f7970dc0bb7fcf3de024ba0432711d946 not found: ID does not exist"
Jul 28 02:04:34.686662 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.686641    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"]
Jul 28 02:04:34.701024 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.698049    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-52pc9"]
Jul 28 02:04:34.755279 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.755244    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:04:34.755279 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.755273    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:04:34.755480 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.755289    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:04:34.755480 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.755301    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdgd9\" (UniqueName: \"kubernetes.io/projected/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-kube-api-access-wdgd9\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:04:34.755480 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:34.755314    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:04:35.057263 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:35.057212    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:35.059728 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:35.059702    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:35.218604 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:35.218508    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:04:35.358005 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:35.357925    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"]
Jul 28 02:04:35.361192 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:04:35.361163    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8732fd8c_41c4_4ad8_8f70_dcd4660b3b49.slice/crio-2c05bbec614b9034d332da70e2ef58c4532234ad95f106dad36b90ef6d42c71e WatchSource:0}: Error finding container 2c05bbec614b9034d332da70e2ef58c4532234ad95f106dad36b90ef6d42c71e: Status 404 returned error can't find the container with id 2c05bbec614b9034d332da70e2ef58c4532234ad95f106dad36b90ef6d42c71e
Jul 28 02:04:35.451267 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:35.451223    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" path="/var/lib/kubelet/pods/ae6e4975-e39f-411c-bab4-22f5dbd8ebc2/volumes"
Jul 28 02:04:35.664643 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:35.664543    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f" event={"ID":"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49","Type":"ContainerStarted","Data":"09d71d044dea352f773119f248b0ab1336f118c6d39871f48532c9854b925b64"}
Jul 28 02:04:35.664643 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:35.664602    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f" event={"ID":"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49","Type":"ContainerStarted","Data":"2c05bbec614b9034d332da70e2ef58c4532234ad95f106dad36b90ef6d42c71e"}
Jul 28 02:04:39.679169 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:39.679138    2555 generic.go:356] "Generic (PLEG): container finished" podID="8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" containerID="09d71d044dea352f773119f248b0ab1336f118c6d39871f48532c9854b925b64" exitCode=0
Jul 28 02:04:39.679563 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:04:39.679173    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f" event={"ID":"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49","Type":"ContainerDied","Data":"09d71d044dea352f773119f248b0ab1336f118c6d39871f48532c9854b925b64"}
Jul 28 02:06:45.156876 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:06:45.156839    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f" event={"ID":"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49","Type":"ContainerStarted","Data":"25125ff96e3efa2fda762648de1158391167bc7150f4650f34a3537da1205354"}
Jul 28 02:06:45.157303 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:06:45.156930    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f" event={"ID":"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49","Type":"ContainerStarted","Data":"a8b7b04a5ced317107e16869d1f22fb2d8c8f80e05af55e7de77f5146394b38a"}
Jul 28 02:06:45.157303 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:06:45.157055    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:06:45.186659 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:06:45.186610    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f" podStartSLOduration=2.585448476 podStartE2EDuration="2m11.186594137s" totalImagesPullingTime="2m4.920126695s" totalInitContainerRuntime="3.681018966s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:06:45.184821376 +0000 UTC m=+2452.194339801" watchObservedRunningTime="2026-07-28 02:06:45.186594137 +0000 UTC m=+2452.196112565"
Jul 28 02:06:46.160424 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:06:46.160388    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:06:52.169720 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:06:52.169687    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:07:22.173937 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:22.173906    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:07:24.452695 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:24.452665    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"]
Jul 28 02:07:24.453134 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:24.452935    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f" podUID="8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" containerName="kserve-container" containerID="cri-o://a8b7b04a5ced317107e16869d1f22fb2d8c8f80e05af55e7de77f5146394b38a" gracePeriod=30
Jul 28 02:07:24.453134 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:24.452978    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f" podUID="8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" containerName="kube-rbac-proxy" containerID="cri-o://25125ff96e3efa2fda762648de1158391167bc7150f4650f34a3537da1205354" gracePeriod=30
Jul 28 02:07:25.294355 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.294324    2555 generic.go:356] "Generic (PLEG): container finished" podID="8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" containerID="25125ff96e3efa2fda762648de1158391167bc7150f4650f34a3537da1205354" exitCode=2
Jul 28 02:07:25.294355 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.294348    2555 generic.go:356] "Generic (PLEG): container finished" podID="8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" containerID="a8b7b04a5ced317107e16869d1f22fb2d8c8f80e05af55e7de77f5146394b38a" exitCode=0
Jul 28 02:07:25.294546 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.294392    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f" event={"ID":"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49","Type":"ContainerDied","Data":"25125ff96e3efa2fda762648de1158391167bc7150f4650f34a3537da1205354"}
Jul 28 02:07:25.294546 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.294447    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f" event={"ID":"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49","Type":"ContainerDied","Data":"a8b7b04a5ced317107e16869d1f22fb2d8c8f80e05af55e7de77f5146394b38a"}
Jul 28 02:07:25.392923 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.392903    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:07:25.445021 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.444999    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kserve-provision-location\") pod \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") "
Jul 28 02:07:25.445155 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.445036    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kube-rbac-proxy-sar-config\") pod \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") "
Jul 28 02:07:25.445155 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.445066    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-cabundle-cert\") pod \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") "
Jul 28 02:07:25.445155 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.445098    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prvzt\" (UniqueName: \"kubernetes.io/projected/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kube-api-access-prvzt\") pod \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") "
Jul 28 02:07:25.445155 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.445131    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-proxy-tls\") pod \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\" (UID: \"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49\") "
Jul 28 02:07:25.445392 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.445374    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" (UID: "8732fd8c-41c4-4ad8-8f70-dcd4660b3b49"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:07:25.445456 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.445420    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" (UID: "8732fd8c-41c4-4ad8-8f70-dcd4660b3b49"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:07:25.445504 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.445452    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" (UID: "8732fd8c-41c4-4ad8-8f70-dcd4660b3b49"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:07:25.447237 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.447210    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kube-api-access-prvzt" (OuterVolumeSpecName: "kube-api-access-prvzt") pod "8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" (UID: "8732fd8c-41c4-4ad8-8f70-dcd4660b3b49"). InnerVolumeSpecName "kube-api-access-prvzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:07:25.447237 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.447214    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" (UID: "8732fd8c-41c4-4ad8-8f70-dcd4660b3b49"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:07:25.545664 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.545632    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:07:25.545664 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.545660    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prvzt\" (UniqueName: \"kubernetes.io/projected/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kube-api-access-prvzt\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:07:25.546047 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.545675    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:07:25.546047 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.545688    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:07:25.546047 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:25.545698    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:07:26.299947 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:26.299918    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"
Jul 28 02:07:26.300144 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:26.299915    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f" event={"ID":"8732fd8c-41c4-4ad8-8f70-dcd4660b3b49","Type":"ContainerDied","Data":"2c05bbec614b9034d332da70e2ef58c4532234ad95f106dad36b90ef6d42c71e"}
Jul 28 02:07:26.300144 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:26.300054    2555 scope.go:117] "RemoveContainer" containerID="25125ff96e3efa2fda762648de1158391167bc7150f4650f34a3537da1205354"
Jul 28 02:07:26.309072 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:26.309054    2555 scope.go:117] "RemoveContainer" containerID="a8b7b04a5ced317107e16869d1f22fb2d8c8f80e05af55e7de77f5146394b38a"
Jul 28 02:07:26.316385 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:26.316366    2555 scope.go:117] "RemoveContainer" containerID="09d71d044dea352f773119f248b0ab1336f118c6d39871f48532c9854b925b64"
Jul 28 02:07:26.320162 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:26.320141    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"]
Jul 28 02:07:26.324677 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:26.324660    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-njm7f"]
Jul 28 02:07:27.451303 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:27.451263    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" path="/var/lib/kubelet/pods/8732fd8c-41c4-4ad8-8f70-dcd4660b3b49/volumes"
Jul 28 02:07:29.605110 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605077    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"]
Jul 28 02:07:29.605509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605302    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="storage-initializer"
Jul 28 02:07:29.605509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605314    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="storage-initializer"
Jul 28 02:07:29.605509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605326    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kube-rbac-proxy"
Jul 28 02:07:29.605509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605331    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kube-rbac-proxy"
Jul 28 02:07:29.605509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605340    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" containerName="kube-rbac-proxy"
Jul 28 02:07:29.605509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605347    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" containerName="kube-rbac-proxy"
Jul 28 02:07:29.605509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605354    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" containerName="storage-initializer"
Jul 28 02:07:29.605509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605359    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" containerName="storage-initializer"
Jul 28 02:07:29.605509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605369    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kserve-container"
Jul 28 02:07:29.605509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605373    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kserve-container"
Jul 28 02:07:29.605509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605381    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" containerName="kserve-container"
Jul 28 02:07:29.605509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605387    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" containerName="kserve-container"
Jul 28 02:07:29.605509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605421    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" containerName="kserve-container"
Jul 28 02:07:29.605509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605427    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kserve-container"
Jul 28 02:07:29.605509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605434    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae6e4975-e39f-411c-bab4-22f5dbd8ebc2" containerName="kube-rbac-proxy"
Jul 28 02:07:29.605509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.605441    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8732fd8c-41c4-4ad8-8f70-dcd4660b3b49" containerName="kube-rbac-proxy"
Jul 28 02:07:29.608291 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.608275    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:29.610702 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.610668    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jul 28 02:07:29.610899 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.610884    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:07:29.610987 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.610932    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:07:29.611052 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.610992    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:07:29.611052 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.611025    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:07:29.611159 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.611050    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 02:07:29.611790 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.611777    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:07:29.617918 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.617895    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"]
Jul 28 02:07:29.669629 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.669602    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07590843-ed71-4868-8766-323d7529313b-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:29.669741 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.669636    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppm27\" (UniqueName: \"kubernetes.io/projected/07590843-ed71-4868-8766-323d7529313b-kube-api-access-ppm27\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:29.669741 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.669656    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07590843-ed71-4868-8766-323d7529313b-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:29.669741 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.669674    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07590843-ed71-4868-8766-323d7529313b-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:29.669741 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.669716    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07590843-ed71-4868-8766-323d7529313b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:29.770394 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.770365    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppm27\" (UniqueName: \"kubernetes.io/projected/07590843-ed71-4868-8766-323d7529313b-kube-api-access-ppm27\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:29.770506 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.770404    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07590843-ed71-4868-8766-323d7529313b-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:29.770506 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.770426    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07590843-ed71-4868-8766-323d7529313b-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:29.770506 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.770460    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07590843-ed71-4868-8766-323d7529313b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:29.770692 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.770543    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07590843-ed71-4868-8766-323d7529313b-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:29.771018 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.770995    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07590843-ed71-4868-8766-323d7529313b-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:29.771231 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.771213    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07590843-ed71-4868-8766-323d7529313b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:29.771289 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.771212    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07590843-ed71-4868-8766-323d7529313b-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:29.772915 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.772898    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07590843-ed71-4868-8766-323d7529313b-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:29.778385 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.778364    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppm27\" (UniqueName: \"kubernetes.io/projected/07590843-ed71-4868-8766-323d7529313b-kube-api-access-ppm27\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:29.918654 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:29.918578    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:30.047154 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:30.047118    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"]
Jul 28 02:07:30.050724 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:07:30.050692    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07590843_ed71_4868_8766_323d7529313b.slice/crio-e6d9da2d743abc3dbff2a0a893e3ccde2a3b6409d18dfe74788cb97ea17f8398 WatchSource:0}: Error finding container e6d9da2d743abc3dbff2a0a893e3ccde2a3b6409d18dfe74788cb97ea17f8398: Status 404 returned error can't find the container with id e6d9da2d743abc3dbff2a0a893e3ccde2a3b6409d18dfe74788cb97ea17f8398
Jul 28 02:07:30.317118 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:30.317072    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489" event={"ID":"07590843-ed71-4868-8766-323d7529313b","Type":"ContainerStarted","Data":"6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e"}
Jul 28 02:07:30.317310 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:30.317162    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489" event={"ID":"07590843-ed71-4868-8766-323d7529313b","Type":"ContainerStarted","Data":"e6d9da2d743abc3dbff2a0a893e3ccde2a3b6409d18dfe74788cb97ea17f8398"}
Jul 28 02:07:34.332291 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:34.332260    2555 generic.go:356] "Generic (PLEG): container finished" podID="07590843-ed71-4868-8766-323d7529313b" containerID="6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e" exitCode=0
Jul 28 02:07:34.332678 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:34.332298    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489" event={"ID":"07590843-ed71-4868-8766-323d7529313b","Type":"ContainerDied","Data":"6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e"}
Jul 28 02:07:35.338423 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:35.338384    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489" event={"ID":"07590843-ed71-4868-8766-323d7529313b","Type":"ContainerStarted","Data":"7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193"}
Jul 28 02:07:35.338839 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:35.338467    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489" event={"ID":"07590843-ed71-4868-8766-323d7529313b","Type":"ContainerStarted","Data":"a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b"}
Jul 28 02:07:35.338839 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:35.338716    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:35.358834 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:35.358787    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489" podStartSLOduration=2.651510549 podStartE2EDuration="6.358770725s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.707260176s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:07:35.358166443 +0000 UTC m=+2502.367684884" watchObservedRunningTime="2026-07-28 02:07:35.358770725 +0000 UTC m=+2502.368289152"
Jul 28 02:07:36.344103 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:36.344070    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:36.345745 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:36.345708    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489" podUID="07590843-ed71-4868-8766-323d7529313b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 02:07:37.347916 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:37.347879    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489" podUID="07590843-ed71-4868-8766-323d7529313b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 02:07:42.352998 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:42.352955    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:42.354211 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:42.354191    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:49.634723 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:49.630565    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"]
Jul 28 02:07:49.635140 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:49.634669    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489" podUID="07590843-ed71-4868-8766-323d7529313b" containerName="kserve-container" containerID="cri-o://a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b" gracePeriod=30
Jul 28 02:07:49.635500 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:49.635156    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489" podUID="07590843-ed71-4868-8766-323d7529313b" containerName="kube-rbac-proxy" containerID="cri-o://7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193" gracePeriod=30
Jul 28 02:07:50.279268 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.279240    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:50.315706 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.315678    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07590843-ed71-4868-8766-323d7529313b-proxy-tls\") pod \"07590843-ed71-4868-8766-323d7529313b\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") "
Jul 28 02:07:50.315873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.315729    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07590843-ed71-4868-8766-323d7529313b-kserve-provision-location\") pod \"07590843-ed71-4868-8766-323d7529313b\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") "
Jul 28 02:07:50.315873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.315764    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07590843-ed71-4868-8766-323d7529313b-cabundle-cert\") pod \"07590843-ed71-4868-8766-323d7529313b\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") "
Jul 28 02:07:50.315873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.315804    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppm27\" (UniqueName: \"kubernetes.io/projected/07590843-ed71-4868-8766-323d7529313b-kube-api-access-ppm27\") pod \"07590843-ed71-4868-8766-323d7529313b\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") "
Jul 28 02:07:50.315873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.315836    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07590843-ed71-4868-8766-323d7529313b-kube-rbac-proxy-sar-config\") pod \"07590843-ed71-4868-8766-323d7529313b\" (UID: \"07590843-ed71-4868-8766-323d7529313b\") "
Jul 28 02:07:50.316199 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.316176    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07590843-ed71-4868-8766-323d7529313b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "07590843-ed71-4868-8766-323d7529313b" (UID: "07590843-ed71-4868-8766-323d7529313b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:07:50.316287 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.316191    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07590843-ed71-4868-8766-323d7529313b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "07590843-ed71-4868-8766-323d7529313b" (UID: "07590843-ed71-4868-8766-323d7529313b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:07:50.316340 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.316298    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07590843-ed71-4868-8766-323d7529313b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "07590843-ed71-4868-8766-323d7529313b" (UID: "07590843-ed71-4868-8766-323d7529313b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:07:50.318095 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.318075    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07590843-ed71-4868-8766-323d7529313b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "07590843-ed71-4868-8766-323d7529313b" (UID: "07590843-ed71-4868-8766-323d7529313b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:07:50.318260 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.318231    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07590843-ed71-4868-8766-323d7529313b-kube-api-access-ppm27" (OuterVolumeSpecName: "kube-api-access-ppm27") pod "07590843-ed71-4868-8766-323d7529313b" (UID: "07590843-ed71-4868-8766-323d7529313b"). InnerVolumeSpecName "kube-api-access-ppm27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:07:50.395685 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.395601    2555 generic.go:356] "Generic (PLEG): container finished" podID="07590843-ed71-4868-8766-323d7529313b" containerID="7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193" exitCode=2
Jul 28 02:07:50.395685 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.395632    2555 generic.go:356] "Generic (PLEG): container finished" podID="07590843-ed71-4868-8766-323d7529313b" containerID="a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b" exitCode=0
Jul 28 02:07:50.395685 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.395665    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489" event={"ID":"07590843-ed71-4868-8766-323d7529313b","Type":"ContainerDied","Data":"7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193"}
Jul 28 02:07:50.395906 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.395718    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489" event={"ID":"07590843-ed71-4868-8766-323d7529313b","Type":"ContainerDied","Data":"a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b"}
Jul 28 02:07:50.395906 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.395738    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489" event={"ID":"07590843-ed71-4868-8766-323d7529313b","Type":"ContainerDied","Data":"e6d9da2d743abc3dbff2a0a893e3ccde2a3b6409d18dfe74788cb97ea17f8398"}
Jul 28 02:07:50.395906 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.395665    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"
Jul 28 02:07:50.395906 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.395763    2555 scope.go:117] "RemoveContainer" containerID="7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193"
Jul 28 02:07:50.404102 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.404079    2555 scope.go:117] "RemoveContainer" containerID="a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b"
Jul 28 02:07:50.411083 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.411060    2555 scope.go:117] "RemoveContainer" containerID="6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e"
Jul 28 02:07:50.416452 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.416426    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07590843-ed71-4868-8766-323d7529313b-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:07:50.416572 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.416456    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07590843-ed71-4868-8766-323d7529313b-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:07:50.416572 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.416466    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07590843-ed71-4868-8766-323d7529313b-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:07:50.416572 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.416475    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppm27\" (UniqueName: \"kubernetes.io/projected/07590843-ed71-4868-8766-323d7529313b-kube-api-access-ppm27\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:07:50.416572 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.416485    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07590843-ed71-4868-8766-323d7529313b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:07:50.417489 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.417468    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"]
Jul 28 02:07:50.420975 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.420954    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vq489"]
Jul 28 02:07:50.421690 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.421662    2555 scope.go:117] "RemoveContainer" containerID="7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193"
Jul 28 02:07:50.421951 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:07:50.421932    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193\": container with ID starting with 7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193 not found: ID does not exist" containerID="7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193"
Jul 28 02:07:50.421996 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.421960    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193"} err="failed to get container status \"7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193\": rpc error: code = NotFound desc = could not find container \"7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193\": container with ID starting with 7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193 not found: ID does not exist"
Jul 28 02:07:50.421996 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.421980    2555 scope.go:117] "RemoveContainer" containerID="a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b"
Jul 28 02:07:50.422228 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:07:50.422202    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b\": container with ID starting with a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b not found: ID does not exist" containerID="a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b"
Jul 28 02:07:50.422305 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.422230    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b"} err="failed to get container status \"a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b\": rpc error: code = NotFound desc = could not find container \"a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b\": container with ID starting with a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b not found: ID does not exist"
Jul 28 02:07:50.422305 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.422253    2555 scope.go:117] "RemoveContainer" containerID="6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e"
Jul 28 02:07:50.422513 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:07:50.422494    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e\": container with ID starting with 6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e not found: ID does not exist" containerID="6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e"
Jul 28 02:07:50.422641 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.422543    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e"} err="failed to get container status \"6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e\": rpc error: code = NotFound desc = could not find container \"6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e\": container with ID starting with 6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e not found: ID does not exist"
Jul 28 02:07:50.422641 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.422559    2555 scope.go:117] "RemoveContainer" containerID="7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193"
Jul 28 02:07:50.422795 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.422776    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193"} err="failed to get container status \"7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193\": rpc error: code = NotFound desc = could not find container \"7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193\": container with ID starting with 7c6e57da646db876ef67e615c96c9827a4eb1002f4d22e958604f68658584193 not found: ID does not exist"
Jul 28 02:07:50.422862 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.422796    2555 scope.go:117] "RemoveContainer" containerID="a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b"
Jul 28 02:07:50.423018 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.423000    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b"} err="failed to get container status \"a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b\": rpc error: code = NotFound desc = could not find container \"a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b\": container with ID starting with a43090a1276b4395e77c31861849919b3b5c35386c3a4be049a7a5b587280c0b not found: ID does not exist"
Jul 28 02:07:50.423069 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.423019    2555 scope.go:117] "RemoveContainer" containerID="6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e"
Jul 28 02:07:50.423272 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:50.423254    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e"} err="failed to get container status \"6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e\": rpc error: code = NotFound desc = could not find container \"6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e\": container with ID starting with 6b0e49857725e01fd62f124fd3c01fc2746aef4d006e92049280cb8cf1200c6e not found: ID does not exist"
Jul 28 02:07:51.451267 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:51.451229    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07590843-ed71-4868-8766-323d7529313b" path="/var/lib/kubelet/pods/07590843-ed71-4868-8766-323d7529313b/volumes"
Jul 28 02:07:54.782737 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.782705    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"]
Jul 28 02:07:54.783116 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.782942    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07590843-ed71-4868-8766-323d7529313b" containerName="kserve-container"
Jul 28 02:07:54.783116 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.782952    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="07590843-ed71-4868-8766-323d7529313b" containerName="kserve-container"
Jul 28 02:07:54.783116 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.782962    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07590843-ed71-4868-8766-323d7529313b" containerName="kube-rbac-proxy"
Jul 28 02:07:54.783116 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.782967    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="07590843-ed71-4868-8766-323d7529313b" containerName="kube-rbac-proxy"
Jul 28 02:07:54.783116 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.782980    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07590843-ed71-4868-8766-323d7529313b" containerName="storage-initializer"
Jul 28 02:07:54.783116 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.782985    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="07590843-ed71-4868-8766-323d7529313b" containerName="storage-initializer"
Jul 28 02:07:54.783116 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.783023    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="07590843-ed71-4868-8766-323d7529313b" containerName="kserve-container"
Jul 28 02:07:54.783116 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.783032    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="07590843-ed71-4868-8766-323d7529313b" containerName="kube-rbac-proxy"
Jul 28 02:07:54.786357 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.786339    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:54.788927 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.788904    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 02:07:54.789157 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.789129    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:07:54.789254 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.789162    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 28 02:07:54.789254 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.789243    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:07:54.789347 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.789325    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:07:54.790093 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.790079    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:07:54.790183 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.790147    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:07:54.798302 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.798284    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"]
Jul 28 02:07:54.845893 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.845871    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf090fc-088d-42a2-af68-640a7ee6f3be-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:54.846011 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.845920    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf090fc-088d-42a2-af68-640a7ee6f3be-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:54.846011 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.845938    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cf090fc-088d-42a2-af68-640a7ee6f3be-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:54.846102 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.846018    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf6gg\" (UniqueName: \"kubernetes.io/projected/5cf090fc-088d-42a2-af68-640a7ee6f3be-kube-api-access-rf6gg\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:54.846102 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.846053    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cf090fc-088d-42a2-af68-640a7ee6f3be-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:54.947218 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.947190    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf090fc-088d-42a2-af68-640a7ee6f3be-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:54.947313 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.947221    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cf090fc-088d-42a2-af68-640a7ee6f3be-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:54.947313 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.947255    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf6gg\" (UniqueName: \"kubernetes.io/projected/5cf090fc-088d-42a2-af68-640a7ee6f3be-kube-api-access-rf6gg\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:54.947313 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.947283    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cf090fc-088d-42a2-af68-640a7ee6f3be-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:54.947441 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.947309    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf090fc-088d-42a2-af68-640a7ee6f3be-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:54.947441 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:07:54.947417    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 28 02:07:54.947553 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:07:54.947483    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf090fc-088d-42a2-af68-640a7ee6f3be-proxy-tls podName:5cf090fc-088d-42a2-af68-640a7ee6f3be nodeName:}" failed. No retries permitted until 2026-07-28 02:07:55.447464449 +0000 UTC m=+2522.456982862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5cf090fc-088d-42a2-af68-640a7ee6f3be-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm" (UID: "5cf090fc-088d-42a2-af68-640a7ee6f3be") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 28 02:07:54.947643 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.947626    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cf090fc-088d-42a2-af68-640a7ee6f3be-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:54.948008 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.947991    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cf090fc-088d-42a2-af68-640a7ee6f3be-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:54.948050 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.948000    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf090fc-088d-42a2-af68-640a7ee6f3be-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:54.957000 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:54.956972    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf6gg\" (UniqueName: \"kubernetes.io/projected/5cf090fc-088d-42a2-af68-640a7ee6f3be-kube-api-access-rf6gg\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:55.450963 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:55.450930    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf090fc-088d-42a2-af68-640a7ee6f3be-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:55.453330 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:55.453308    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf090fc-088d-42a2-af68-640a7ee6f3be-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:55.696009 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:55.695966    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:07:55.821472 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:55.821444    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"]
Jul 28 02:07:55.825181 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:07:55.825156    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf090fc_088d_42a2_af68_640a7ee6f3be.slice/crio-508363c6755ca45421255e36c1e5fe64091acde8a56fe8a629c3150724fddb56 WatchSource:0}: Error finding container 508363c6755ca45421255e36c1e5fe64091acde8a56fe8a629c3150724fddb56: Status 404 returned error can't find the container with id 508363c6755ca45421255e36c1e5fe64091acde8a56fe8a629c3150724fddb56
Jul 28 02:07:56.416924 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:56.416891    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm" event={"ID":"5cf090fc-088d-42a2-af68-640a7ee6f3be","Type":"ContainerStarted","Data":"75e71f41c920b276340bb8e57f11da9d093ff86daaf1cfac8093563a635c5e74"}
Jul 28 02:07:56.417091 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:07:56.416949    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm" event={"ID":"5cf090fc-088d-42a2-af68-640a7ee6f3be","Type":"ContainerStarted","Data":"508363c6755ca45421255e36c1e5fe64091acde8a56fe8a629c3150724fddb56"}
Jul 28 02:08:00.432208 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:00.432176    2555 generic.go:356] "Generic (PLEG): container finished" podID="5cf090fc-088d-42a2-af68-640a7ee6f3be" containerID="75e71f41c920b276340bb8e57f11da9d093ff86daaf1cfac8093563a635c5e74" exitCode=0
Jul 28 02:08:00.432603 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:00.432248    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm" event={"ID":"5cf090fc-088d-42a2-af68-640a7ee6f3be","Type":"ContainerDied","Data":"75e71f41c920b276340bb8e57f11da9d093ff86daaf1cfac8093563a635c5e74"}
Jul 28 02:08:01.438282 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:01.438241    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm" event={"ID":"5cf090fc-088d-42a2-af68-640a7ee6f3be","Type":"ContainerStarted","Data":"310167a09a40b12edc956bd1926df66b3ded6dbf14357f378861074e391ae452"}
Jul 28 02:08:01.438699 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:01.438339    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm" event={"ID":"5cf090fc-088d-42a2-af68-640a7ee6f3be","Type":"ContainerStarted","Data":"b3087efffde66b1a8742c1ef505d3d0b5f4cf904088f3c92504a0284bd9c2818"}
Jul 28 02:08:01.438699 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:01.438683    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:08:01.438785 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:01.438728    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:08:01.462089 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:01.462031    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm" podStartSLOduration=3.623919302 podStartE2EDuration="7.462013288s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.838093986s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:08:01.461781262 +0000 UTC m=+2528.471299687" watchObservedRunningTime="2026-07-28 02:08:01.462013288 +0000 UTC m=+2528.471531714"
Jul 28 02:08:07.450718 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:07.450691    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:08:37.454622 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:37.454579    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:08:44.956878 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:44.956840    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"]
Jul 28 02:08:44.957283 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:44.957064    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm" podUID="5cf090fc-088d-42a2-af68-640a7ee6f3be" containerName="kserve-container" containerID="cri-o://b3087efffde66b1a8742c1ef505d3d0b5f4cf904088f3c92504a0284bd9c2818" gracePeriod=30
Jul 28 02:08:44.957283 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:44.957104    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm" podUID="5cf090fc-088d-42a2-af68-640a7ee6f3be" containerName="kube-rbac-proxy" containerID="cri-o://310167a09a40b12edc956bd1926df66b3ded6dbf14357f378861074e391ae452" gracePeriod=30
Jul 28 02:08:45.581516 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:45.581483    2555 generic.go:356] "Generic (PLEG): container finished" podID="5cf090fc-088d-42a2-af68-640a7ee6f3be" containerID="310167a09a40b12edc956bd1926df66b3ded6dbf14357f378861074e391ae452" exitCode=2
Jul 28 02:08:45.581690 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:45.581552    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm" event={"ID":"5cf090fc-088d-42a2-af68-640a7ee6f3be","Type":"ContainerDied","Data":"310167a09a40b12edc956bd1926df66b3ded6dbf14357f378861074e391ae452"}
Jul 28 02:08:46.094354 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.094331    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:08:46.185790 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.185706    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf6gg\" (UniqueName: \"kubernetes.io/projected/5cf090fc-088d-42a2-af68-640a7ee6f3be-kube-api-access-rf6gg\") pod \"5cf090fc-088d-42a2-af68-640a7ee6f3be\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") "
Jul 28 02:08:46.185790 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.185753    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cf090fc-088d-42a2-af68-640a7ee6f3be-cabundle-cert\") pod \"5cf090fc-088d-42a2-af68-640a7ee6f3be\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") "
Jul 28 02:08:46.185790 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.185786    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cf090fc-088d-42a2-af68-640a7ee6f3be-kserve-provision-location\") pod \"5cf090fc-088d-42a2-af68-640a7ee6f3be\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") "
Jul 28 02:08:46.186037 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.185836    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf090fc-088d-42a2-af68-640a7ee6f3be-proxy-tls\") pod \"5cf090fc-088d-42a2-af68-640a7ee6f3be\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") "
Jul 28 02:08:46.186037 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.185877    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf090fc-088d-42a2-af68-640a7ee6f3be-kube-rbac-proxy-sar-config\") pod \"5cf090fc-088d-42a2-af68-640a7ee6f3be\" (UID: \"5cf090fc-088d-42a2-af68-640a7ee6f3be\") "
Jul 28 02:08:46.186216 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.186184    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf090fc-088d-42a2-af68-640a7ee6f3be-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5cf090fc-088d-42a2-af68-640a7ee6f3be" (UID: "5cf090fc-088d-42a2-af68-640a7ee6f3be"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:08:46.186297 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.186260    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf090fc-088d-42a2-af68-640a7ee6f3be-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5cf090fc-088d-42a2-af68-640a7ee6f3be" (UID: "5cf090fc-088d-42a2-af68-640a7ee6f3be"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:08:46.186339 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.186309    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf090fc-088d-42a2-af68-640a7ee6f3be-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5cf090fc-088d-42a2-af68-640a7ee6f3be" (UID: "5cf090fc-088d-42a2-af68-640a7ee6f3be"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:08:46.188019 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.188002    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cf090fc-088d-42a2-af68-640a7ee6f3be-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5cf090fc-088d-42a2-af68-640a7ee6f3be" (UID: "5cf090fc-088d-42a2-af68-640a7ee6f3be"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:08:46.188145 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.188127    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf090fc-088d-42a2-af68-640a7ee6f3be-kube-api-access-rf6gg" (OuterVolumeSpecName: "kube-api-access-rf6gg") pod "5cf090fc-088d-42a2-af68-640a7ee6f3be" (UID: "5cf090fc-088d-42a2-af68-640a7ee6f3be"). InnerVolumeSpecName "kube-api-access-rf6gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:08:46.286461 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.286425    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf090fc-088d-42a2-af68-640a7ee6f3be-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:08:46.286461 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.286454    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf090fc-088d-42a2-af68-640a7ee6f3be-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:08:46.286461 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.286466    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf6gg\" (UniqueName: \"kubernetes.io/projected/5cf090fc-088d-42a2-af68-640a7ee6f3be-kube-api-access-rf6gg\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:08:46.286729 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.286479    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cf090fc-088d-42a2-af68-640a7ee6f3be-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:08:46.286729 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.286491    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cf090fc-088d-42a2-af68-640a7ee6f3be-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:08:46.587077 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.587047    2555 generic.go:356] "Generic (PLEG): container finished" podID="5cf090fc-088d-42a2-af68-640a7ee6f3be" containerID="b3087efffde66b1a8742c1ef505d3d0b5f4cf904088f3c92504a0284bd9c2818" exitCode=0
Jul 28 02:08:46.587242 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.587107    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"
Jul 28 02:08:46.587242 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.587114    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm" event={"ID":"5cf090fc-088d-42a2-af68-640a7ee6f3be","Type":"ContainerDied","Data":"b3087efffde66b1a8742c1ef505d3d0b5f4cf904088f3c92504a0284bd9c2818"}
Jul 28 02:08:46.587242 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.587160    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm" event={"ID":"5cf090fc-088d-42a2-af68-640a7ee6f3be","Type":"ContainerDied","Data":"508363c6755ca45421255e36c1e5fe64091acde8a56fe8a629c3150724fddb56"}
Jul 28 02:08:46.587242 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.587186    2555 scope.go:117] "RemoveContainer" containerID="310167a09a40b12edc956bd1926df66b3ded6dbf14357f378861074e391ae452"
Jul 28 02:08:46.595382 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.595365    2555 scope.go:117] "RemoveContainer" containerID="b3087efffde66b1a8742c1ef505d3d0b5f4cf904088f3c92504a0284bd9c2818"
Jul 28 02:08:46.602566 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.602545    2555 scope.go:117] "RemoveContainer" containerID="75e71f41c920b276340bb8e57f11da9d093ff86daaf1cfac8093563a635c5e74"
Jul 28 02:08:46.609456 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.609437    2555 scope.go:117] "RemoveContainer" containerID="310167a09a40b12edc956bd1926df66b3ded6dbf14357f378861074e391ae452"
Jul 28 02:08:46.609709 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:08:46.609690    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310167a09a40b12edc956bd1926df66b3ded6dbf14357f378861074e391ae452\": container with ID starting with 310167a09a40b12edc956bd1926df66b3ded6dbf14357f378861074e391ae452 not found: ID does not exist" containerID="310167a09a40b12edc956bd1926df66b3ded6dbf14357f378861074e391ae452"
Jul 28 02:08:46.609759 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.609720    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310167a09a40b12edc956bd1926df66b3ded6dbf14357f378861074e391ae452"} err="failed to get container status \"310167a09a40b12edc956bd1926df66b3ded6dbf14357f378861074e391ae452\": rpc error: code = NotFound desc = could not find container \"310167a09a40b12edc956bd1926df66b3ded6dbf14357f378861074e391ae452\": container with ID starting with 310167a09a40b12edc956bd1926df66b3ded6dbf14357f378861074e391ae452 not found: ID does not exist"
Jul 28 02:08:46.609759 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.609746    2555 scope.go:117] "RemoveContainer" containerID="b3087efffde66b1a8742c1ef505d3d0b5f4cf904088f3c92504a0284bd9c2818"
Jul 28 02:08:46.609985 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:08:46.609969    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3087efffde66b1a8742c1ef505d3d0b5f4cf904088f3c92504a0284bd9c2818\": container with ID starting with b3087efffde66b1a8742c1ef505d3d0b5f4cf904088f3c92504a0284bd9c2818 not found: ID does not exist" containerID="b3087efffde66b1a8742c1ef505d3d0b5f4cf904088f3c92504a0284bd9c2818"
Jul 28 02:08:46.610022 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.609992    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3087efffde66b1a8742c1ef505d3d0b5f4cf904088f3c92504a0284bd9c2818"} err="failed to get container status \"b3087efffde66b1a8742c1ef505d3d0b5f4cf904088f3c92504a0284bd9c2818\": rpc error: code = NotFound desc = could not find container \"b3087efffde66b1a8742c1ef505d3d0b5f4cf904088f3c92504a0284bd9c2818\": container with ID starting with b3087efffde66b1a8742c1ef505d3d0b5f4cf904088f3c92504a0284bd9c2818 not found: ID does not exist"
Jul 28 02:08:46.610022 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.610007    2555 scope.go:117] "RemoveContainer" containerID="75e71f41c920b276340bb8e57f11da9d093ff86daaf1cfac8093563a635c5e74"
Jul 28 02:08:46.610206 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:08:46.610190    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75e71f41c920b276340bb8e57f11da9d093ff86daaf1cfac8093563a635c5e74\": container with ID starting with 75e71f41c920b276340bb8e57f11da9d093ff86daaf1cfac8093563a635c5e74 not found: ID does not exist" containerID="75e71f41c920b276340bb8e57f11da9d093ff86daaf1cfac8093563a635c5e74"
Jul 28 02:08:46.610244 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.610210    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75e71f41c920b276340bb8e57f11da9d093ff86daaf1cfac8093563a635c5e74"} err="failed to get container status \"75e71f41c920b276340bb8e57f11da9d093ff86daaf1cfac8093563a635c5e74\": rpc error: code = NotFound desc = could not find container \"75e71f41c920b276340bb8e57f11da9d093ff86daaf1cfac8093563a635c5e74\": container with ID starting with 75e71f41c920b276340bb8e57f11da9d093ff86daaf1cfac8093563a635c5e74 not found: ID does not exist"
Jul 28 02:08:46.612172 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.612151    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"]
Jul 28 02:08:46.617403 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:46.617384    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6bgvm"]
Jul 28 02:08:47.451626 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:47.451586    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cf090fc-088d-42a2-af68-640a7ee6f3be" path="/var/lib/kubelet/pods/5cf090fc-088d-42a2-af68-640a7ee6f3be/volumes"
Jul 28 02:08:50.098738 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.098697    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"]
Jul 28 02:08:50.099189 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.099037    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf090fc-088d-42a2-af68-640a7ee6f3be" containerName="kserve-container"
Jul 28 02:08:50.099189 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.099056    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf090fc-088d-42a2-af68-640a7ee6f3be" containerName="kserve-container"
Jul 28 02:08:50.099189 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.099075    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf090fc-088d-42a2-af68-640a7ee6f3be" containerName="kube-rbac-proxy"
Jul 28 02:08:50.099189 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.099082    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf090fc-088d-42a2-af68-640a7ee6f3be" containerName="kube-rbac-proxy"
Jul 28 02:08:50.099189 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.099107    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf090fc-088d-42a2-af68-640a7ee6f3be" containerName="storage-initializer"
Jul 28 02:08:50.099189 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.099115    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf090fc-088d-42a2-af68-640a7ee6f3be" containerName="storage-initializer"
Jul 28 02:08:50.099189 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.099190    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf090fc-088d-42a2-af68-640a7ee6f3be" containerName="kube-rbac-proxy"
Jul 28 02:08:50.099598 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.099202    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf090fc-088d-42a2-af68-640a7ee6f3be" containerName="kserve-container"
Jul 28 02:08:50.102732 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.102709    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:50.105731 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.105696    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:08:50.105851 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.105809    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:08:50.105950 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.105931    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 28 02:08:50.106849 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.106830    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 28 02:08:50.106948 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.106846    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:08:50.106996 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.106955    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:08:50.107057 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.107044    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:08:50.113203 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.113177    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"]
Jul 28 02:08:50.213992 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.213958    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab530cce-ebe1-4956-9f06-36de919e8835-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-b4cx9\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:50.214159 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.214021    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab530cce-ebe1-4956-9f06-36de919e8835-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-b4cx9\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:50.214159 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.214056    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9qq8\" (UniqueName: \"kubernetes.io/projected/ab530cce-ebe1-4956-9f06-36de919e8835-kube-api-access-j9qq8\") pod \"isvc-sklearn-mcp-predictor-5996957568-b4cx9\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:50.214159 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.214082    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab530cce-ebe1-4956-9f06-36de919e8835-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-b4cx9\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:50.214159 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.214100    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab530cce-ebe1-4956-9f06-36de919e8835-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-b4cx9\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:50.314860 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.314823    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab530cce-ebe1-4956-9f06-36de919e8835-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-b4cx9\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:50.315039 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.314869    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9qq8\" (UniqueName: \"kubernetes.io/projected/ab530cce-ebe1-4956-9f06-36de919e8835-kube-api-access-j9qq8\") pod \"isvc-sklearn-mcp-predictor-5996957568-b4cx9\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:50.315039 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.314904    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab530cce-ebe1-4956-9f06-36de919e8835-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-b4cx9\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:50.315039 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.314932    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab530cce-ebe1-4956-9f06-36de919e8835-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-b4cx9\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:50.315039 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.314964    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab530cce-ebe1-4956-9f06-36de919e8835-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-b4cx9\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:50.315039 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:08:50.314986    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Jul 28 02:08:50.315283 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:08:50.315062    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab530cce-ebe1-4956-9f06-36de919e8835-proxy-tls podName:ab530cce-ebe1-4956-9f06-36de919e8835 nodeName:}" failed. No retries permitted until 2026-07-28 02:08:50.815044704 +0000 UTC m=+2577.824563108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ab530cce-ebe1-4956-9f06-36de919e8835-proxy-tls") pod "isvc-sklearn-mcp-predictor-5996957568-b4cx9" (UID: "ab530cce-ebe1-4956-9f06-36de919e8835") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Jul 28 02:08:50.315415 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.315386    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab530cce-ebe1-4956-9f06-36de919e8835-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-b4cx9\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:50.315737 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.315718    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab530cce-ebe1-4956-9f06-36de919e8835-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-b4cx9\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:50.315838 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.315789    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab530cce-ebe1-4956-9f06-36de919e8835-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-b4cx9\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:50.324458 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.324432    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9qq8\" (UniqueName: \"kubernetes.io/projected/ab530cce-ebe1-4956-9f06-36de919e8835-kube-api-access-j9qq8\") pod \"isvc-sklearn-mcp-predictor-5996957568-b4cx9\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:50.817311 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.817277    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab530cce-ebe1-4956-9f06-36de919e8835-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-b4cx9\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:50.819763 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:50.819730    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab530cce-ebe1-4956-9f06-36de919e8835-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-b4cx9\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:51.013640 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:51.013606    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:51.147648 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:51.147621    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"]
Jul 28 02:08:51.152622 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:08:51.150650    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab530cce_ebe1_4956_9f06_36de919e8835.slice/crio-b4d9c4155d90c244cf2d9dd49d03ded388cabcd00d8628d1be68d1c558d99f67 WatchSource:0}: Error finding container b4d9c4155d90c244cf2d9dd49d03ded388cabcd00d8628d1be68d1c558d99f67: Status 404 returned error can't find the container with id b4d9c4155d90c244cf2d9dd49d03ded388cabcd00d8628d1be68d1c558d99f67
Jul 28 02:08:51.610940 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:51.610911    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" event={"ID":"ab530cce-ebe1-4956-9f06-36de919e8835","Type":"ContainerStarted","Data":"02f012086d342938cbf001157c2b0f9ddccf025f42597fb500e4a96aee5fcbd1"}
Jul 28 02:08:51.611106 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:51.610983    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" event={"ID":"ab530cce-ebe1-4956-9f06-36de919e8835","Type":"ContainerStarted","Data":"b4d9c4155d90c244cf2d9dd49d03ded388cabcd00d8628d1be68d1c558d99f67"}
Jul 28 02:08:55.625030 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:55.624995    2555 generic.go:356] "Generic (PLEG): container finished" podID="ab530cce-ebe1-4956-9f06-36de919e8835" containerID="02f012086d342938cbf001157c2b0f9ddccf025f42597fb500e4a96aee5fcbd1" exitCode=0
Jul 28 02:08:55.625401 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:55.625052    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" event={"ID":"ab530cce-ebe1-4956-9f06-36de919e8835","Type":"ContainerDied","Data":"02f012086d342938cbf001157c2b0f9ddccf025f42597fb500e4a96aee5fcbd1"}
Jul 28 02:08:55.681445 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:55.681428    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 02:08:56.629734 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:56.629695    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" event={"ID":"ab530cce-ebe1-4956-9f06-36de919e8835","Type":"ContainerStarted","Data":"9f49c264cdfaca76c68aeac6ed431ee4c9782dae2b07510bd581926c86f9fde3"}
Jul 28 02:08:58.638536 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:58.638497    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" event={"ID":"ab530cce-ebe1-4956-9f06-36de919e8835","Type":"ContainerStarted","Data":"37330dc93db191326cc4ab242bd99ba03abe9bb5a99566c002330efc531be713"}
Jul 28 02:08:58.638919 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:58.638586    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" event={"ID":"ab530cce-ebe1-4956-9f06-36de919e8835","Type":"ContainerStarted","Data":"e1bf66859b16e67a6030444595c06b788a0e44283c5543a55d3f9c9df3d147d3"}
Jul 28 02:08:58.638919 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:58.638695    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:58.638919 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:58.638843    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:58.638919 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:58.638902    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:08:58.665100 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:08:58.665047    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" podStartSLOduration=2.539254279 podStartE2EDuration="8.665027461s" totalImagesPullingTime="2.352827963s" totalInitContainerRuntime="3.772945219s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:08:58.662272319 +0000 UTC m=+2585.671790743" watchObservedRunningTime="2026-07-28 02:08:58.665027461 +0000 UTC m=+2585.674545886"
Jul 28 02:09:04.646888 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:09:04.646857    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:09:34.649131 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:09:34.649105    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:10:04.650346 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:04.650308    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:10:10.150952 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:10.150918    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"]
Jul 28 02:10:10.151404 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:10.151186    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kserve-container" containerID="cri-o://9f49c264cdfaca76c68aeac6ed431ee4c9782dae2b07510bd581926c86f9fde3" gracePeriod=30
Jul 28 02:10:10.151404 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:10.151237    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kserve-agent" containerID="cri-o://e1bf66859b16e67a6030444595c06b788a0e44283c5543a55d3f9c9df3d147d3" gracePeriod=30
Jul 28 02:10:10.151404 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:10.151237    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kube-rbac-proxy" containerID="cri-o://37330dc93db191326cc4ab242bd99ba03abe9bb5a99566c002330efc531be713" gracePeriod=30
Jul 28 02:10:10.889383 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:10.889352    2555 generic.go:356] "Generic (PLEG): container finished" podID="ab530cce-ebe1-4956-9f06-36de919e8835" containerID="37330dc93db191326cc4ab242bd99ba03abe9bb5a99566c002330efc531be713" exitCode=2
Jul 28 02:10:10.889601 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:10.889419    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" event={"ID":"ab530cce-ebe1-4956-9f06-36de919e8835","Type":"ContainerDied","Data":"37330dc93db191326cc4ab242bd99ba03abe9bb5a99566c002330efc531be713"}
Jul 28 02:10:12.898518 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:12.898483    2555 generic.go:356] "Generic (PLEG): container finished" podID="ab530cce-ebe1-4956-9f06-36de919e8835" containerID="9f49c264cdfaca76c68aeac6ed431ee4c9782dae2b07510bd581926c86f9fde3" exitCode=0
Jul 28 02:10:12.898927 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:12.898563    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" event={"ID":"ab530cce-ebe1-4956-9f06-36de919e8835","Type":"ContainerDied","Data":"9f49c264cdfaca76c68aeac6ed431ee4c9782dae2b07510bd581926c86f9fde3"}
Jul 28 02:10:14.642475 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:14.642426    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 28 02:10:14.647989 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:14.647963    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.39:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 02:10:15.274391 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.274345    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"]
Jul 28 02:10:15.276687 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.276664    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:15.279296 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.279266    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jul 28 02:10:15.279296 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.279284    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jul 28 02:10:15.288099 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.288078    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"]
Jul 28 02:10:15.312001 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.311972    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gk8ww\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:15.312146 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.312009    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a824873-ca4a-4bf8-bc45-ce35bd5af556-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gk8ww\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:15.312146 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.312032    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gk8ww\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:15.312146 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.312094    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a824873-ca4a-4bf8-bc45-ce35bd5af556-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gk8ww\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:15.312289 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.312152    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxwlc\" (UniqueName: \"kubernetes.io/projected/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kube-api-access-sxwlc\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gk8ww\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:15.412604 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.412569    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gk8ww\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:15.412799 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.412611    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a824873-ca4a-4bf8-bc45-ce35bd5af556-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gk8ww\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:15.412799 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.412632    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gk8ww\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:15.412799 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.412667    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a824873-ca4a-4bf8-bc45-ce35bd5af556-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gk8ww\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:15.412799 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.412705    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwlc\" (UniqueName: \"kubernetes.io/projected/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kube-api-access-sxwlc\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gk8ww\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:15.413023 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:10:15.412835    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Jul 28 02:10:15.413023 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:10:15.412909    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a824873-ca4a-4bf8-bc45-ce35bd5af556-proxy-tls podName:5a824873-ca4a-4bf8-bc45-ce35bd5af556 nodeName:}" failed. No retries permitted until 2026-07-28 02:10:15.912888881 +0000 UTC m=+2662.922407298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5a824873-ca4a-4bf8-bc45-ce35bd5af556-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-gk8ww" (UID: "5a824873-ca4a-4bf8-bc45-ce35bd5af556") : secret "isvc-paddle-predictor-serving-cert" not found
Jul 28 02:10:15.413128 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.413024    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gk8ww\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:15.413370 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.413350    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gk8ww\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:15.413411 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.413360    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a824873-ca4a-4bf8-bc45-ce35bd5af556-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gk8ww\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:15.421875 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.421850    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxwlc\" (UniqueName: \"kubernetes.io/projected/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kube-api-access-sxwlc\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gk8ww\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:15.918384 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.918353    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a824873-ca4a-4bf8-bc45-ce35bd5af556-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gk8ww\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:15.920806 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:15.920781    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a824873-ca4a-4bf8-bc45-ce35bd5af556-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gk8ww\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:16.186318 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:16.186223    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:16.315264 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:16.315238    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"]
Jul 28 02:10:16.317661 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:10:16.317634    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a824873_ca4a_4bf8_bc45_ce35bd5af556.slice/crio-1a24c97bbf0762cccab97d03015399a8bea526ef552c1ae7d3c40487276e59d1 WatchSource:0}: Error finding container 1a24c97bbf0762cccab97d03015399a8bea526ef552c1ae7d3c40487276e59d1: Status 404 returned error can't find the container with id 1a24c97bbf0762cccab97d03015399a8bea526ef552c1ae7d3c40487276e59d1
Jul 28 02:10:16.914979 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:16.914944    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" event={"ID":"5a824873-ca4a-4bf8-bc45-ce35bd5af556","Type":"ContainerStarted","Data":"ae6f530f9247f7b8ed61c13d31129f41a0954e7530f60000df762b636e487148"}
Jul 28 02:10:16.915157 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:16.915008    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" event={"ID":"5a824873-ca4a-4bf8-bc45-ce35bd5af556","Type":"ContainerStarted","Data":"1a24c97bbf0762cccab97d03015399a8bea526ef552c1ae7d3c40487276e59d1"}
Jul 28 02:10:19.642642 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:19.642593    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 28 02:10:21.935776 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:21.935721    2555 generic.go:356] "Generic (PLEG): container finished" podID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerID="ae6f530f9247f7b8ed61c13d31129f41a0954e7530f60000df762b636e487148" exitCode=0
Jul 28 02:10:21.935776 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:21.935778    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" event={"ID":"5a824873-ca4a-4bf8-bc45-ce35bd5af556","Type":"ContainerDied","Data":"ae6f530f9247f7b8ed61c13d31129f41a0954e7530f60000df762b636e487148"}
Jul 28 02:10:24.641922 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:24.641875    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 28 02:10:24.642378 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:24.642011    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:10:24.648179 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:24.648142    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.39:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 02:10:29.642841 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:29.642789    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 28 02:10:33.989368 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:33.989330    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" event={"ID":"5a824873-ca4a-4bf8-bc45-ce35bd5af556","Type":"ContainerStarted","Data":"80b927d030383edfdc0a11be06f29b62ecf245997159f65cba3c8bde93eaf5d3"}
Jul 28 02:10:33.989772 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:33.989402    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" event={"ID":"5a824873-ca4a-4bf8-bc45-ce35bd5af556","Type":"ContainerStarted","Data":"8fd8dc8c96528c9314e1eecb1340e114c77314f99f9ee7a78ef4d6fc77dd5102"}
Jul 28 02:10:33.989772 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:33.989611    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:34.011360 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:34.011305    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" podStartSLOduration=2.814322613 podStartE2EDuration="19.011290167s" totalImagesPullingTime="11.637498299s" totalInitContainerRuntime="4.559469255s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:10:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:10:34.009931747 +0000 UTC m=+2681.019450172" watchObservedRunningTime="2026-07-28 02:10:34.011290167 +0000 UTC m=+2681.020808591"
Jul 28 02:10:34.642377 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:34.642325    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 28 02:10:34.647782 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:34.647751    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.39:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 02:10:34.647876 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:34.647866    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:10:34.992458 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:34.992429    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:34.993598 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:34.993560    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 02:10:35.996053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:35.996011    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 02:10:39.641918 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:39.641874    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 28 02:10:40.301038 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:40.301011    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:10:40.417558 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:40.417431    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab530cce-ebe1-4956-9f06-36de919e8835-kserve-provision-location\") pod \"ab530cce-ebe1-4956-9f06-36de919e8835\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") "
Jul 28 02:10:40.417558 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:40.417473    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab530cce-ebe1-4956-9f06-36de919e8835-proxy-tls\") pod \"ab530cce-ebe1-4956-9f06-36de919e8835\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") "
Jul 28 02:10:40.417558 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:40.417498    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab530cce-ebe1-4956-9f06-36de919e8835-kube-rbac-proxy-sar-config\") pod \"ab530cce-ebe1-4956-9f06-36de919e8835\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") "
Jul 28 02:10:40.417558 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:40.417542    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9qq8\" (UniqueName: \"kubernetes.io/projected/ab530cce-ebe1-4956-9f06-36de919e8835-kube-api-access-j9qq8\") pod \"ab530cce-ebe1-4956-9f06-36de919e8835\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") "
Jul 28 02:10:40.417851 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:40.417595    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab530cce-ebe1-4956-9f06-36de919e8835-cabundle-cert\") pod \"ab530cce-ebe1-4956-9f06-36de919e8835\" (UID: \"ab530cce-ebe1-4956-9f06-36de919e8835\") "
Jul 28 02:10:40.417898 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:40.417865    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab530cce-ebe1-4956-9f06-36de919e8835-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab530cce-ebe1-4956-9f06-36de919e8835" (UID: "ab530cce-ebe1-4956-9f06-36de919e8835"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:10:40.418038 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:40.418008    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab530cce-ebe1-4956-9f06-36de919e8835-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ab530cce-ebe1-4956-9f06-36de919e8835" (UID: "ab530cce-ebe1-4956-9f06-36de919e8835"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:10:40.418098 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:40.418057    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab530cce-ebe1-4956-9f06-36de919e8835-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ab530cce-ebe1-4956-9f06-36de919e8835" (UID: "ab530cce-ebe1-4956-9f06-36de919e8835"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:10:40.419958 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:40.419929    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab530cce-ebe1-4956-9f06-36de919e8835-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab530cce-ebe1-4956-9f06-36de919e8835" (UID: "ab530cce-ebe1-4956-9f06-36de919e8835"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:10:40.419958 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:40.419940    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab530cce-ebe1-4956-9f06-36de919e8835-kube-api-access-j9qq8" (OuterVolumeSpecName: "kube-api-access-j9qq8") pod "ab530cce-ebe1-4956-9f06-36de919e8835" (UID: "ab530cce-ebe1-4956-9f06-36de919e8835"). InnerVolumeSpecName "kube-api-access-j9qq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:10:40.519070 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:40.519015    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab530cce-ebe1-4956-9f06-36de919e8835-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:10:40.519070 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:40.519065    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab530cce-ebe1-4956-9f06-36de919e8835-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:10:40.519070 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:40.519080    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab530cce-ebe1-4956-9f06-36de919e8835-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:10:40.519353 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:40.519095    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9qq8\" (UniqueName: \"kubernetes.io/projected/ab530cce-ebe1-4956-9f06-36de919e8835-kube-api-access-j9qq8\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:10:40.519353 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:40.519108    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab530cce-ebe1-4956-9f06-36de919e8835-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:10:41.000846 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.000814    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:10:41.001402 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.001375    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 02:10:41.015321 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.015286    2555 generic.go:356] "Generic (PLEG): container finished" podID="ab530cce-ebe1-4956-9f06-36de919e8835" containerID="e1bf66859b16e67a6030444595c06b788a0e44283c5543a55d3f9c9df3d147d3" exitCode=0
Jul 28 02:10:41.015499 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.015352    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" event={"ID":"ab530cce-ebe1-4956-9f06-36de919e8835","Type":"ContainerDied","Data":"e1bf66859b16e67a6030444595c06b788a0e44283c5543a55d3f9c9df3d147d3"}
Jul 28 02:10:41.015499 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.015357    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"
Jul 28 02:10:41.015499 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.015407    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9" event={"ID":"ab530cce-ebe1-4956-9f06-36de919e8835","Type":"ContainerDied","Data":"b4d9c4155d90c244cf2d9dd49d03ded388cabcd00d8628d1be68d1c558d99f67"}
Jul 28 02:10:41.015499 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.015436    2555 scope.go:117] "RemoveContainer" containerID="37330dc93db191326cc4ab242bd99ba03abe9bb5a99566c002330efc531be713"
Jul 28 02:10:41.025314 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.025296    2555 scope.go:117] "RemoveContainer" containerID="e1bf66859b16e67a6030444595c06b788a0e44283c5543a55d3f9c9df3d147d3"
Jul 28 02:10:41.032833 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.032814    2555 scope.go:117] "RemoveContainer" containerID="9f49c264cdfaca76c68aeac6ed431ee4c9782dae2b07510bd581926c86f9fde3"
Jul 28 02:10:41.038183 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.038151    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"]
Jul 28 02:10:41.040479 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.040455    2555 scope.go:117] "RemoveContainer" containerID="02f012086d342938cbf001157c2b0f9ddccf025f42597fb500e4a96aee5fcbd1"
Jul 28 02:10:41.042323 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.042304    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-b4cx9"]
Jul 28 02:10:41.048329 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.048309    2555 scope.go:117] "RemoveContainer" containerID="37330dc93db191326cc4ab242bd99ba03abe9bb5a99566c002330efc531be713"
Jul 28 02:10:41.048675 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:10:41.048655    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37330dc93db191326cc4ab242bd99ba03abe9bb5a99566c002330efc531be713\": container with ID starting with 37330dc93db191326cc4ab242bd99ba03abe9bb5a99566c002330efc531be713 not found: ID does not exist" containerID="37330dc93db191326cc4ab242bd99ba03abe9bb5a99566c002330efc531be713"
Jul 28 02:10:41.048767 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.048690    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37330dc93db191326cc4ab242bd99ba03abe9bb5a99566c002330efc531be713"} err="failed to get container status \"37330dc93db191326cc4ab242bd99ba03abe9bb5a99566c002330efc531be713\": rpc error: code = NotFound desc = could not find container \"37330dc93db191326cc4ab242bd99ba03abe9bb5a99566c002330efc531be713\": container with ID starting with 37330dc93db191326cc4ab242bd99ba03abe9bb5a99566c002330efc531be713 not found: ID does not exist"
Jul 28 02:10:41.048767 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.048719    2555 scope.go:117] "RemoveContainer" containerID="e1bf66859b16e67a6030444595c06b788a0e44283c5543a55d3f9c9df3d147d3"
Jul 28 02:10:41.048971 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:10:41.048953    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1bf66859b16e67a6030444595c06b788a0e44283c5543a55d3f9c9df3d147d3\": container with ID starting with e1bf66859b16e67a6030444595c06b788a0e44283c5543a55d3f9c9df3d147d3 not found: ID does not exist" containerID="e1bf66859b16e67a6030444595c06b788a0e44283c5543a55d3f9c9df3d147d3"
Jul 28 02:10:41.049020 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.048977    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bf66859b16e67a6030444595c06b788a0e44283c5543a55d3f9c9df3d147d3"} err="failed to get container status \"e1bf66859b16e67a6030444595c06b788a0e44283c5543a55d3f9c9df3d147d3\": rpc error: code = NotFound desc = could not find container \"e1bf66859b16e67a6030444595c06b788a0e44283c5543a55d3f9c9df3d147d3\": container with ID starting with e1bf66859b16e67a6030444595c06b788a0e44283c5543a55d3f9c9df3d147d3 not found: ID does not exist"
Jul 28 02:10:41.049020 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.048993    2555 scope.go:117] "RemoveContainer" containerID="9f49c264cdfaca76c68aeac6ed431ee4c9782dae2b07510bd581926c86f9fde3"
Jul 28 02:10:41.049274 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:10:41.049257    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f49c264cdfaca76c68aeac6ed431ee4c9782dae2b07510bd581926c86f9fde3\": container with ID starting with 9f49c264cdfaca76c68aeac6ed431ee4c9782dae2b07510bd581926c86f9fde3 not found: ID does not exist" containerID="9f49c264cdfaca76c68aeac6ed431ee4c9782dae2b07510bd581926c86f9fde3"
Jul 28 02:10:41.049331 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.049281    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f49c264cdfaca76c68aeac6ed431ee4c9782dae2b07510bd581926c86f9fde3"} err="failed to get container status \"9f49c264cdfaca76c68aeac6ed431ee4c9782dae2b07510bd581926c86f9fde3\": rpc error: code = NotFound desc = could not find container \"9f49c264cdfaca76c68aeac6ed431ee4c9782dae2b07510bd581926c86f9fde3\": container with ID starting with 9f49c264cdfaca76c68aeac6ed431ee4c9782dae2b07510bd581926c86f9fde3 not found: ID does not exist"
Jul 28 02:10:41.049331 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.049301    2555 scope.go:117] "RemoveContainer" containerID="02f012086d342938cbf001157c2b0f9ddccf025f42597fb500e4a96aee5fcbd1"
Jul 28 02:10:41.049635 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:10:41.049511    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f012086d342938cbf001157c2b0f9ddccf025f42597fb500e4a96aee5fcbd1\": container with ID starting with 02f012086d342938cbf001157c2b0f9ddccf025f42597fb500e4a96aee5fcbd1 not found: ID does not exist" containerID="02f012086d342938cbf001157c2b0f9ddccf025f42597fb500e4a96aee5fcbd1"
Jul 28 02:10:41.049684 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.049642    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f012086d342938cbf001157c2b0f9ddccf025f42597fb500e4a96aee5fcbd1"} err="failed to get container status \"02f012086d342938cbf001157c2b0f9ddccf025f42597fb500e4a96aee5fcbd1\": rpc error: code = NotFound desc = could not find container \"02f012086d342938cbf001157c2b0f9ddccf025f42597fb500e4a96aee5fcbd1\": container with ID starting with 02f012086d342938cbf001157c2b0f9ddccf025f42597fb500e4a96aee5fcbd1 not found: ID does not exist"
Jul 28 02:10:41.452016 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:41.451936    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" path="/var/lib/kubelet/pods/ab530cce-ebe1-4956-9f06-36de919e8835/volumes"
Jul 28 02:10:51.001444 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:10:51.001356    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 02:11:01.001737 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:01.001693    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 02:11:11.001447 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:11.001403    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 02:11:21.002181 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:21.002149    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:11:27.033789 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:27.033753    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"]
Jul 28 02:11:27.034247 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:27.034022    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerName="kserve-container" containerID="cri-o://8fd8dc8c96528c9314e1eecb1340e114c77314f99f9ee7a78ef4d6fc77dd5102" gracePeriod=30
Jul 28 02:11:27.034247 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:27.034055    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerName="kube-rbac-proxy" containerID="cri-o://80b927d030383edfdc0a11be06f29b62ecf245997159f65cba3c8bde93eaf5d3" gracePeriod=30
Jul 28 02:11:27.183335 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:27.183303    2555 generic.go:356] "Generic (PLEG): container finished" podID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerID="80b927d030383edfdc0a11be06f29b62ecf245997159f65cba3c8bde93eaf5d3" exitCode=2
Jul 28 02:11:27.183335 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:27.183346    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" event={"ID":"5a824873-ca4a-4bf8-bc45-ce35bd5af556","Type":"ContainerDied","Data":"80b927d030383edfdc0a11be06f29b62ecf245997159f65cba3c8bde93eaf5d3"}
Jul 28 02:11:29.982780 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:29.982751    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:11:30.089831 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.089738    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxwlc\" (UniqueName: \"kubernetes.io/projected/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kube-api-access-sxwlc\") pod \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") "
Jul 28 02:11:30.089831 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.089782    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a824873-ca4a-4bf8-bc45-ce35bd5af556-proxy-tls\") pod \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") "
Jul 28 02:11:30.089831 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.089821    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a824873-ca4a-4bf8-bc45-ce35bd5af556-cabundle-cert\") pod \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") "
Jul 28 02:11:30.090109 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.089865    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kserve-provision-location\") pod \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") "
Jul 28 02:11:30.090109 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.089890    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kube-rbac-proxy-sar-config\") pod \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\" (UID: \"5a824873-ca4a-4bf8-bc45-ce35bd5af556\") "
Jul 28 02:11:30.090269 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.090242    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a824873-ca4a-4bf8-bc45-ce35bd5af556-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5a824873-ca4a-4bf8-bc45-ce35bd5af556" (UID: "5a824873-ca4a-4bf8-bc45-ce35bd5af556"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:11:30.090365 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.090336    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a824873-ca4a-4bf8-bc45-ce35bd5af556" (UID: "5a824873-ca4a-4bf8-bc45-ce35bd5af556"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:11:30.092232 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.092207    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a824873-ca4a-4bf8-bc45-ce35bd5af556-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a824873-ca4a-4bf8-bc45-ce35bd5af556" (UID: "5a824873-ca4a-4bf8-bc45-ce35bd5af556"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:11:30.092343 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.092243    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kube-api-access-sxwlc" (OuterVolumeSpecName: "kube-api-access-sxwlc") pod "5a824873-ca4a-4bf8-bc45-ce35bd5af556" (UID: "5a824873-ca4a-4bf8-bc45-ce35bd5af556"). InnerVolumeSpecName "kube-api-access-sxwlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:11:30.099181 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.099145    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a824873-ca4a-4bf8-bc45-ce35bd5af556" (UID: "5a824873-ca4a-4bf8-bc45-ce35bd5af556"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:11:30.190885 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.190850    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxwlc\" (UniqueName: \"kubernetes.io/projected/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kube-api-access-sxwlc\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:11:30.190885 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.190878    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a824873-ca4a-4bf8-bc45-ce35bd5af556-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:11:30.190885 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.190889    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a824873-ca4a-4bf8-bc45-ce35bd5af556-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:11:30.191132 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.190897    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:11:30.191132 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.190908    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a824873-ca4a-4bf8-bc45-ce35bd5af556-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:11:30.195376 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.195349    2555 generic.go:356] "Generic (PLEG): container finished" podID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerID="8fd8dc8c96528c9314e1eecb1340e114c77314f99f9ee7a78ef4d6fc77dd5102" exitCode=0
Jul 28 02:11:30.195503 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.195390    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" event={"ID":"5a824873-ca4a-4bf8-bc45-ce35bd5af556","Type":"ContainerDied","Data":"8fd8dc8c96528c9314e1eecb1340e114c77314f99f9ee7a78ef4d6fc77dd5102"}
Jul 28 02:11:30.195503 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.195404    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"
Jul 28 02:11:30.195503 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.195444    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww" event={"ID":"5a824873-ca4a-4bf8-bc45-ce35bd5af556","Type":"ContainerDied","Data":"1a24c97bbf0762cccab97d03015399a8bea526ef552c1ae7d3c40487276e59d1"}
Jul 28 02:11:30.195503 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.195471    2555 scope.go:117] "RemoveContainer" containerID="80b927d030383edfdc0a11be06f29b62ecf245997159f65cba3c8bde93eaf5d3"
Jul 28 02:11:30.203925 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.203908    2555 scope.go:117] "RemoveContainer" containerID="8fd8dc8c96528c9314e1eecb1340e114c77314f99f9ee7a78ef4d6fc77dd5102"
Jul 28 02:11:30.211398 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.211378    2555 scope.go:117] "RemoveContainer" containerID="ae6f530f9247f7b8ed61c13d31129f41a0954e7530f60000df762b636e487148"
Jul 28 02:11:30.217188 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.217166    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"]
Jul 28 02:11:30.218940 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.218919    2555 scope.go:117] "RemoveContainer" containerID="80b927d030383edfdc0a11be06f29b62ecf245997159f65cba3c8bde93eaf5d3"
Jul 28 02:11:30.219246 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:11:30.219220    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b927d030383edfdc0a11be06f29b62ecf245997159f65cba3c8bde93eaf5d3\": container with ID starting with 80b927d030383edfdc0a11be06f29b62ecf245997159f65cba3c8bde93eaf5d3 not found: ID does not exist" containerID="80b927d030383edfdc0a11be06f29b62ecf245997159f65cba3c8bde93eaf5d3"
Jul 28 02:11:30.219323 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.219258    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b927d030383edfdc0a11be06f29b62ecf245997159f65cba3c8bde93eaf5d3"} err="failed to get container status \"80b927d030383edfdc0a11be06f29b62ecf245997159f65cba3c8bde93eaf5d3\": rpc error: code = NotFound desc = could not find container \"80b927d030383edfdc0a11be06f29b62ecf245997159f65cba3c8bde93eaf5d3\": container with ID starting with 80b927d030383edfdc0a11be06f29b62ecf245997159f65cba3c8bde93eaf5d3 not found: ID does not exist"
Jul 28 02:11:30.219323 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.219284    2555 scope.go:117] "RemoveContainer" containerID="8fd8dc8c96528c9314e1eecb1340e114c77314f99f9ee7a78ef4d6fc77dd5102"
Jul 28 02:11:30.219626 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:11:30.219587    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd8dc8c96528c9314e1eecb1340e114c77314f99f9ee7a78ef4d6fc77dd5102\": container with ID starting with 8fd8dc8c96528c9314e1eecb1340e114c77314f99f9ee7a78ef4d6fc77dd5102 not found: ID does not exist" containerID="8fd8dc8c96528c9314e1eecb1340e114c77314f99f9ee7a78ef4d6fc77dd5102"
Jul 28 02:11:30.219729 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.219647    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd8dc8c96528c9314e1eecb1340e114c77314f99f9ee7a78ef4d6fc77dd5102"} err="failed to get container status \"8fd8dc8c96528c9314e1eecb1340e114c77314f99f9ee7a78ef4d6fc77dd5102\": rpc error: code = NotFound desc = could not find container \"8fd8dc8c96528c9314e1eecb1340e114c77314f99f9ee7a78ef4d6fc77dd5102\": container with ID starting with 8fd8dc8c96528c9314e1eecb1340e114c77314f99f9ee7a78ef4d6fc77dd5102 not found: ID does not exist"
Jul 28 02:11:30.219729 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.219672    2555 scope.go:117] "RemoveContainer" containerID="ae6f530f9247f7b8ed61c13d31129f41a0954e7530f60000df762b636e487148"
Jul 28 02:11:30.220040 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:11:30.220009    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6f530f9247f7b8ed61c13d31129f41a0954e7530f60000df762b636e487148\": container with ID starting with ae6f530f9247f7b8ed61c13d31129f41a0954e7530f60000df762b636e487148 not found: ID does not exist" containerID="ae6f530f9247f7b8ed61c13d31129f41a0954e7530f60000df762b636e487148"
Jul 28 02:11:30.220128 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.220042    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6f530f9247f7b8ed61c13d31129f41a0954e7530f60000df762b636e487148"} err="failed to get container status \"ae6f530f9247f7b8ed61c13d31129f41a0954e7530f60000df762b636e487148\": rpc error: code = NotFound desc = could not find container \"ae6f530f9247f7b8ed61c13d31129f41a0954e7530f60000df762b636e487148\": container with ID starting with ae6f530f9247f7b8ed61c13d31129f41a0954e7530f60000df762b636e487148 not found: ID does not exist"
Jul 28 02:11:30.221264 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:30.221242    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gk8ww"]
Jul 28 02:11:31.451827 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:31.451790    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" path="/var/lib/kubelet/pods/5a824873-ca4a-4bf8-bc45-ce35bd5af556/volumes"
Jul 28 02:11:32.193466 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193429    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"]
Jul 28 02:11:32.193758 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193710    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerName="storage-initializer"
Jul 28 02:11:32.193758 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193723    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerName="storage-initializer"
Jul 28 02:11:32.193758 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193734    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kserve-agent"
Jul 28 02:11:32.193758 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193740    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kserve-agent"
Jul 28 02:11:32.193758 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193746    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kube-rbac-proxy"
Jul 28 02:11:32.193758 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193752    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kube-rbac-proxy"
Jul 28 02:11:32.193758 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193761    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="storage-initializer"
Jul 28 02:11:32.194014 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193766    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="storage-initializer"
Jul 28 02:11:32.194014 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193774    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kserve-container"
Jul 28 02:11:32.194014 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193780    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kserve-container"
Jul 28 02:11:32.194014 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193788    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerName="kserve-container"
Jul 28 02:11:32.194014 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193793    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerName="kserve-container"
Jul 28 02:11:32.194014 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193799    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerName="kube-rbac-proxy"
Jul 28 02:11:32.194014 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193805    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerName="kube-rbac-proxy"
Jul 28 02:11:32.194014 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193850    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerName="kserve-container"
Jul 28 02:11:32.194014 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193856    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kube-rbac-proxy"
Jul 28 02:11:32.194014 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193862    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kserve-agent"
Jul 28 02:11:32.194014 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193866    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a824873-ca4a-4bf8-bc45-ce35bd5af556" containerName="kube-rbac-proxy"
Jul 28 02:11:32.194014 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.193871    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab530cce-ebe1-4956-9f06-36de919e8835" containerName="kserve-container"
Jul 28 02:11:32.197336 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.197318    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:32.200152 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.200125    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:11:32.200152 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.200126    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 28 02:11:32.200330 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.200158    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 28 02:11:32.200330 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.200247    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:11:32.200617 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.200599    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:11:32.200712 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.200653    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:11:32.203825 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.203804    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:11:32.208265 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.208241    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"]
Jul 28 02:11:32.307310 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.307272    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mdp45\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:32.307491 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.307325    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mdp45\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:32.307491 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.307345    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m42t5\" (UniqueName: \"kubernetes.io/projected/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kube-api-access-m42t5\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mdp45\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:32.307491 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.307407    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mdp45\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:32.307491 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.307458    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mdp45\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:32.408095 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.408057    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mdp45\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:32.408199 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.408098    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m42t5\" (UniqueName: \"kubernetes.io/projected/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kube-api-access-m42t5\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mdp45\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:32.408199 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.408134    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mdp45\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:32.408199 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.408177    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mdp45\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:32.408346 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.408204    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mdp45\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:32.408408 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:11:32.408336    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 28 02:11:32.408460 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:11:32.408413    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-proxy-tls podName:18a58bb7-8c63-41f8-ae3d-80a776d7eebb nodeName:}" failed. No retries permitted until 2026-07-28 02:11:32.908389728 +0000 UTC m=+2739.917908144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-mdp45" (UID: "18a58bb7-8c63-41f8-ae3d-80a776d7eebb") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 28 02:11:32.408687 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.408668    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mdp45\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:32.408864 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.408847    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mdp45\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:32.408904 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.408862    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mdp45\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:32.417025 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.417002    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m42t5\" (UniqueName: \"kubernetes.io/projected/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kube-api-access-m42t5\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mdp45\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:32.913401 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.913359    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mdp45\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:32.915892 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:32.915870    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mdp45\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:33.108337 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:33.108295    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:33.240412 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:33.240377    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"]
Jul 28 02:11:33.243447 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:11:33.243421    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a58bb7_8c63_41f8_ae3d_80a776d7eebb.slice/crio-93c41d6b4241a9b5bc1789cee460a8377b5be60ec2166e1fdda436df83224084 WatchSource:0}: Error finding container 93c41d6b4241a9b5bc1789cee460a8377b5be60ec2166e1fdda436df83224084: Status 404 returned error can't find the container with id 93c41d6b4241a9b5bc1789cee460a8377b5be60ec2166e1fdda436df83224084
Jul 28 02:11:34.214110 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:34.214069    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" event={"ID":"18a58bb7-8c63-41f8-ae3d-80a776d7eebb","Type":"ContainerStarted","Data":"e7e46261c8ba8e5bf50e33e6efb6e41b4b52f7e66e47e32c2a186218034cd8eb"}
Jul 28 02:11:34.214623 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:34.214129    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" event={"ID":"18a58bb7-8c63-41f8-ae3d-80a776d7eebb","Type":"ContainerStarted","Data":"93c41d6b4241a9b5bc1789cee460a8377b5be60ec2166e1fdda436df83224084"}
Jul 28 02:11:38.229646 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:38.229552    2555 generic.go:356] "Generic (PLEG): container finished" podID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerID="e7e46261c8ba8e5bf50e33e6efb6e41b4b52f7e66e47e32c2a186218034cd8eb" exitCode=0
Jul 28 02:11:38.229646 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:38.229607    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" event={"ID":"18a58bb7-8c63-41f8-ae3d-80a776d7eebb","Type":"ContainerDied","Data":"e7e46261c8ba8e5bf50e33e6efb6e41b4b52f7e66e47e32c2a186218034cd8eb"}
Jul 28 02:11:39.235282 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:39.235246    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" event={"ID":"18a58bb7-8c63-41f8-ae3d-80a776d7eebb","Type":"ContainerStarted","Data":"3ac1cbcff213560b7a47f1a2de1b3720a17eb3daa8212eb6d0437ea4dea64526"}
Jul 28 02:11:39.235745 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:39.235309    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" event={"ID":"18a58bb7-8c63-41f8-ae3d-80a776d7eebb","Type":"ContainerStarted","Data":"fed1dbac4cd0694f894ffbb39fd906bd957a404915f16afe5a97b6a9de133f32"}
Jul 28 02:11:39.235745 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:39.235647    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:39.235854 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:39.235786    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:39.237094 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:39.237072    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 02:11:39.261299 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:39.261243    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" podStartSLOduration=2.777546345 podStartE2EDuration="7.261223999s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.483677654s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:11:39.261093091 +0000 UTC m=+2746.270611537" watchObservedRunningTime="2026-07-28 02:11:39.261223999 +0000 UTC m=+2746.270742424"
Jul 28 02:11:40.238948 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:40.238905    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 02:11:41.242677 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:41.242629    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 02:11:46.246859 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:46.246829    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:11:46.247434 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:46.247401    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 02:11:56.247933 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:11:56.247890    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 02:12:06.247508 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:06.247466    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 02:12:16.248156 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:16.248070    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 02:12:26.248686 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:26.248651    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:12:33.833190 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:33.833157    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"]
Jul 28 02:12:33.833703 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:33.833384    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kserve-container" containerID="cri-o://fed1dbac4cd0694f894ffbb39fd906bd957a404915f16afe5a97b6a9de133f32" gracePeriod=30
Jul 28 02:12:33.833703 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:33.833447    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kube-rbac-proxy" containerID="cri-o://3ac1cbcff213560b7a47f1a2de1b3720a17eb3daa8212eb6d0437ea4dea64526" gracePeriod=30
Jul 28 02:12:34.427487 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:34.427455    2555 generic.go:356] "Generic (PLEG): container finished" podID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerID="3ac1cbcff213560b7a47f1a2de1b3720a17eb3daa8212eb6d0437ea4dea64526" exitCode=2
Jul 28 02:12:34.427668 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:34.427539    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" event={"ID":"18a58bb7-8c63-41f8-ae3d-80a776d7eebb","Type":"ContainerDied","Data":"3ac1cbcff213560b7a47f1a2de1b3720a17eb3daa8212eb6d0437ea4dea64526"}
Jul 28 02:12:36.242924 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.242877    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 28 02:12:36.248318 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.248285    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 02:12:36.437226 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.437195    2555 generic.go:356] "Generic (PLEG): container finished" podID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerID="fed1dbac4cd0694f894ffbb39fd906bd957a404915f16afe5a97b6a9de133f32" exitCode=0
Jul 28 02:12:36.437392 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.437259    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" event={"ID":"18a58bb7-8c63-41f8-ae3d-80a776d7eebb","Type":"ContainerDied","Data":"fed1dbac4cd0694f894ffbb39fd906bd957a404915f16afe5a97b6a9de133f32"}
Jul 28 02:12:36.476061 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.476037    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:12:36.573505 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.573413    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-cabundle-cert\") pod \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") "
Jul 28 02:12:36.573505 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.573452    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kserve-provision-location\") pod \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") "
Jul 28 02:12:36.573505 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.573497    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-proxy-tls\") pod \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") "
Jul 28 02:12:36.573818 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.573512    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kube-rbac-proxy-sar-config\") pod \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") "
Jul 28 02:12:36.573818 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.573701    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m42t5\" (UniqueName: \"kubernetes.io/projected/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kube-api-access-m42t5\") pod \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\" (UID: \"18a58bb7-8c63-41f8-ae3d-80a776d7eebb\") "
Jul 28 02:12:36.573925 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.573893    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "18a58bb7-8c63-41f8-ae3d-80a776d7eebb" (UID: "18a58bb7-8c63-41f8-ae3d-80a776d7eebb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:12:36.573925 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.573900    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "18a58bb7-8c63-41f8-ae3d-80a776d7eebb" (UID: "18a58bb7-8c63-41f8-ae3d-80a776d7eebb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:12:36.575736 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.575709    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "18a58bb7-8c63-41f8-ae3d-80a776d7eebb" (UID: "18a58bb7-8c63-41f8-ae3d-80a776d7eebb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:12:36.575918 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.575890    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kube-api-access-m42t5" (OuterVolumeSpecName: "kube-api-access-m42t5") pod "18a58bb7-8c63-41f8-ae3d-80a776d7eebb" (UID: "18a58bb7-8c63-41f8-ae3d-80a776d7eebb"). InnerVolumeSpecName "kube-api-access-m42t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:12:36.583310 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.583282    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "18a58bb7-8c63-41f8-ae3d-80a776d7eebb" (UID: "18a58bb7-8c63-41f8-ae3d-80a776d7eebb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:12:36.674560 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.674497    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:12:36.674560 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.674555    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:12:36.674560 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.674566    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m42t5\" (UniqueName: \"kubernetes.io/projected/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kube-api-access-m42t5\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:12:36.674560 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.674576    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:12:36.674812 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:36.674584    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18a58bb7-8c63-41f8-ae3d-80a776d7eebb-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:12:37.442031 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:37.441998    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45" event={"ID":"18a58bb7-8c63-41f8-ae3d-80a776d7eebb","Type":"ContainerDied","Data":"93c41d6b4241a9b5bc1789cee460a8377b5be60ec2166e1fdda436df83224084"}
Jul 28 02:12:37.442469 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:37.442048    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"
Jul 28 02:12:37.442469 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:37.442065    2555 scope.go:117] "RemoveContainer" containerID="3ac1cbcff213560b7a47f1a2de1b3720a17eb3daa8212eb6d0437ea4dea64526"
Jul 28 02:12:37.451185 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:37.451161    2555 scope.go:117] "RemoveContainer" containerID="fed1dbac4cd0694f894ffbb39fd906bd957a404915f16afe5a97b6a9de133f32"
Jul 28 02:12:37.458742 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:37.458724    2555 scope.go:117] "RemoveContainer" containerID="e7e46261c8ba8e5bf50e33e6efb6e41b4b52f7e66e47e32c2a186218034cd8eb"
Jul 28 02:12:37.465015 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:37.464991    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"]
Jul 28 02:12:37.471377 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:37.471356    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mdp45"]
Jul 28 02:12:38.997631 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:38.997600    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"]
Jul 28 02:12:38.997997 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:38.997827    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kserve-container"
Jul 28 02:12:38.997997 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:38.997837    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kserve-container"
Jul 28 02:12:38.997997 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:38.997851    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kube-rbac-proxy"
Jul 28 02:12:38.997997 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:38.997857    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kube-rbac-proxy"
Jul 28 02:12:38.997997 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:38.997863    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="storage-initializer"
Jul 28 02:12:38.997997 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:38.997869    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="storage-initializer"
Jul 28 02:12:38.997997 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:38.997911    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kube-rbac-proxy"
Jul 28 02:12:38.997997 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:38.997919    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" containerName="kserve-container"
Jul 28 02:12:39.000878 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.000860    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.003787 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.003765    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 02:12:39.003787 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.003784    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:12:39.003983 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.003823    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:12:39.004074 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.004059    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:12:39.005025 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.005007    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:12:39.005108 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.005009    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 28 02:12:39.005108 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.005058    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:12:39.012240 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.012219    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"]
Jul 28 02:12:39.191139 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.191091    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1fd040f-ee0f-4bf6-b032-fb985659c042-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.191139 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.191143    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1fd040f-ee0f-4bf6-b032-fb985659c042-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.191351 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.191231    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k46rh\" (UniqueName: \"kubernetes.io/projected/e1fd040f-ee0f-4bf6-b032-fb985659c042-kube-api-access-k46rh\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.191351 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.191260    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1fd040f-ee0f-4bf6-b032-fb985659c042-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.191351 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.191306    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1fd040f-ee0f-4bf6-b032-fb985659c042-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.292296 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.292219    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1fd040f-ee0f-4bf6-b032-fb985659c042-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.292296 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.292280    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k46rh\" (UniqueName: \"kubernetes.io/projected/e1fd040f-ee0f-4bf6-b032-fb985659c042-kube-api-access-k46rh\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.292296 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.292300    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1fd040f-ee0f-4bf6-b032-fb985659c042-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.292485 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.292323    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1fd040f-ee0f-4bf6-b032-fb985659c042-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.292485 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.292353    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1fd040f-ee0f-4bf6-b032-fb985659c042-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.292485 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:12:39.292461    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Jul 28 02:12:39.292646 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:12:39.292586    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1fd040f-ee0f-4bf6-b032-fb985659c042-proxy-tls podName:e1fd040f-ee0f-4bf6-b032-fb985659c042 nodeName:}" failed. No retries permitted until 2026-07-28 02:12:39.792561358 +0000 UTC m=+2806.802079763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e1fd040f-ee0f-4bf6-b032-fb985659c042-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" (UID: "e1fd040f-ee0f-4bf6-b032-fb985659c042") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Jul 28 02:12:39.292816 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.292800    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1fd040f-ee0f-4bf6-b032-fb985659c042-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.292946 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.292928    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1fd040f-ee0f-4bf6-b032-fb985659c042-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.293024 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.293005    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1fd040f-ee0f-4bf6-b032-fb985659c042-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.301821 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.301801    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k46rh\" (UniqueName: \"kubernetes.io/projected/e1fd040f-ee0f-4bf6-b032-fb985659c042-kube-api-access-k46rh\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.451810 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.451777    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18a58bb7-8c63-41f8-ae3d-80a776d7eebb" path="/var/lib/kubelet/pods/18a58bb7-8c63-41f8-ae3d-80a776d7eebb/volumes"
Jul 28 02:12:39.795585 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.795512    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1fd040f-ee0f-4bf6-b032-fb985659c042-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.798040 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.798017    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1fd040f-ee0f-4bf6-b032-fb985659c042-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:39.910437 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:39.910384    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:40.036044 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:40.036019    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"]
Jul 28 02:12:40.038446 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:12:40.038413    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1fd040f_ee0f_4bf6_b032_fb985659c042.slice/crio-fbd67ee5e96a7818ca32a3a15e22663c66e63a1ae9bf947909a45c1cf106fc10 WatchSource:0}: Error finding container fbd67ee5e96a7818ca32a3a15e22663c66e63a1ae9bf947909a45c1cf106fc10: Status 404 returned error can't find the container with id fbd67ee5e96a7818ca32a3a15e22663c66e63a1ae9bf947909a45c1cf106fc10
Jul 28 02:12:40.454670 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:40.454582    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" event={"ID":"e1fd040f-ee0f-4bf6-b032-fb985659c042","Type":"ContainerStarted","Data":"090d8580ff2e55e9a7b8116e30e5cc7949e0ed4ad8e86fa3dd7a9a6b6fdaa914"}
Jul 28 02:12:40.454670 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:40.454661    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" event={"ID":"e1fd040f-ee0f-4bf6-b032-fb985659c042","Type":"ContainerStarted","Data":"fbd67ee5e96a7818ca32a3a15e22663c66e63a1ae9bf947909a45c1cf106fc10"}
Jul 28 02:12:44.467862 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:44.467788    2555 generic.go:356] "Generic (PLEG): container finished" podID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerID="090d8580ff2e55e9a7b8116e30e5cc7949e0ed4ad8e86fa3dd7a9a6b6fdaa914" exitCode=0
Jul 28 02:12:44.468195 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:44.467860    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" event={"ID":"e1fd040f-ee0f-4bf6-b032-fb985659c042","Type":"ContainerDied","Data":"090d8580ff2e55e9a7b8116e30e5cc7949e0ed4ad8e86fa3dd7a9a6b6fdaa914"}
Jul 28 02:12:45.474081 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:45.474039    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" event={"ID":"e1fd040f-ee0f-4bf6-b032-fb985659c042","Type":"ContainerStarted","Data":"6db5417e6581e9e22467907de183ef80e0ee0942013cc776e1be5c78da9f66e0"}
Jul 28 02:12:45.474452 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:45.474126    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" event={"ID":"e1fd040f-ee0f-4bf6-b032-fb985659c042","Type":"ContainerStarted","Data":"53ec6007b1a558540dba36ea25b0dd8907be5b208b563c4c60108e76d5fe6475"}
Jul 28 02:12:45.474452 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:45.474345    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:45.496028 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:45.495984    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" podStartSLOduration=3.376671115 podStartE2EDuration="7.495969613s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.119298498s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:12:45.495015991 +0000 UTC m=+2812.504534416" watchObservedRunningTime="2026-07-28 02:12:45.495969613 +0000 UTC m=+2812.505488038"
Jul 28 02:12:46.477771 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:46.477741    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:46.478961 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:46.478934    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 02:12:47.480630 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:47.480585    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 02:12:52.484697 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:52.484668    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:12:52.485209 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:12:52.485181    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 02:13:02.485889 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:02.485839    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 02:13:12.485113 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:12.485072    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 02:13:22.485989 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:22.485945    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 02:13:32.486565 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:32.486459    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:13:40.846694 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:40.846662    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"]
Jul 28 02:13:40.847122 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:40.846911    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="kserve-container" containerID="cri-o://53ec6007b1a558540dba36ea25b0dd8907be5b208b563c4c60108e76d5fe6475" gracePeriod=30
Jul 28 02:13:40.847122 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:40.846930    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="kube-rbac-proxy" containerID="cri-o://6db5417e6581e9e22467907de183ef80e0ee0942013cc776e1be5c78da9f66e0" gracePeriod=30
Jul 28 02:13:41.663894 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:41.663861    2555 generic.go:356] "Generic (PLEG): container finished" podID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerID="6db5417e6581e9e22467907de183ef80e0ee0942013cc776e1be5c78da9f66e0" exitCode=2
Jul 28 02:13:41.664063 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:41.663934    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" event={"ID":"e1fd040f-ee0f-4bf6-b032-fb985659c042","Type":"ContainerDied","Data":"6db5417e6581e9e22467907de183ef80e0ee0942013cc776e1be5c78da9f66e0"}
Jul 28 02:13:42.481692 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:42.481651    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Jul 28 02:13:42.485972 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:42.485949    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 02:13:43.483790 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.483762    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:13:43.551366 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.551299    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1fd040f-ee0f-4bf6-b032-fb985659c042-kube-rbac-proxy-sar-config\") pod \"e1fd040f-ee0f-4bf6-b032-fb985659c042\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") "
Jul 28 02:13:43.551366 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.551333    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1fd040f-ee0f-4bf6-b032-fb985659c042-proxy-tls\") pod \"e1fd040f-ee0f-4bf6-b032-fb985659c042\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") "
Jul 28 02:13:43.551366 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.551363    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1fd040f-ee0f-4bf6-b032-fb985659c042-cabundle-cert\") pod \"e1fd040f-ee0f-4bf6-b032-fb985659c042\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") "
Jul 28 02:13:43.551662 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.551389    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k46rh\" (UniqueName: \"kubernetes.io/projected/e1fd040f-ee0f-4bf6-b032-fb985659c042-kube-api-access-k46rh\") pod \"e1fd040f-ee0f-4bf6-b032-fb985659c042\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") "
Jul 28 02:13:43.551662 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.551417    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1fd040f-ee0f-4bf6-b032-fb985659c042-kserve-provision-location\") pod \"e1fd040f-ee0f-4bf6-b032-fb985659c042\" (UID: \"e1fd040f-ee0f-4bf6-b032-fb985659c042\") "
Jul 28 02:13:43.551783 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.551698    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1fd040f-ee0f-4bf6-b032-fb985659c042-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e1fd040f-ee0f-4bf6-b032-fb985659c042" (UID: "e1fd040f-ee0f-4bf6-b032-fb985659c042"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:13:43.551851 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.551784    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1fd040f-ee0f-4bf6-b032-fb985659c042-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e1fd040f-ee0f-4bf6-b032-fb985659c042" (UID: "e1fd040f-ee0f-4bf6-b032-fb985659c042"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:13:43.553547 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.553507    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1fd040f-ee0f-4bf6-b032-fb985659c042-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1fd040f-ee0f-4bf6-b032-fb985659c042" (UID: "e1fd040f-ee0f-4bf6-b032-fb985659c042"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:13:43.553658 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.553580    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1fd040f-ee0f-4bf6-b032-fb985659c042-kube-api-access-k46rh" (OuterVolumeSpecName: "kube-api-access-k46rh") pod "e1fd040f-ee0f-4bf6-b032-fb985659c042" (UID: "e1fd040f-ee0f-4bf6-b032-fb985659c042"). InnerVolumeSpecName "kube-api-access-k46rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:13:43.566207 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.566179    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fd040f-ee0f-4bf6-b032-fb985659c042-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e1fd040f-ee0f-4bf6-b032-fb985659c042" (UID: "e1fd040f-ee0f-4bf6-b032-fb985659c042"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:13:43.652117 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.652086    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1fd040f-ee0f-4bf6-b032-fb985659c042-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:13:43.652117 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.652111    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1fd040f-ee0f-4bf6-b032-fb985659c042-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:13:43.652395 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.652122    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1fd040f-ee0f-4bf6-b032-fb985659c042-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:13:43.652395 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.652131    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k46rh\" (UniqueName: \"kubernetes.io/projected/e1fd040f-ee0f-4bf6-b032-fb985659c042-kube-api-access-k46rh\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:13:43.652395 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.652140    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1fd040f-ee0f-4bf6-b032-fb985659c042-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:13:43.673288 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.673259    2555 generic.go:356] "Generic (PLEG): container finished" podID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerID="53ec6007b1a558540dba36ea25b0dd8907be5b208b563c4c60108e76d5fe6475" exitCode=0
Jul 28 02:13:43.673384 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.673299    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" event={"ID":"e1fd040f-ee0f-4bf6-b032-fb985659c042","Type":"ContainerDied","Data":"53ec6007b1a558540dba36ea25b0dd8907be5b208b563c4c60108e76d5fe6475"}
Jul 28 02:13:43.673384 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.673317    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"
Jul 28 02:13:43.673384 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.673357    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm" event={"ID":"e1fd040f-ee0f-4bf6-b032-fb985659c042","Type":"ContainerDied","Data":"fbd67ee5e96a7818ca32a3a15e22663c66e63a1ae9bf947909a45c1cf106fc10"}
Jul 28 02:13:43.673487 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.673387    2555 scope.go:117] "RemoveContainer" containerID="6db5417e6581e9e22467907de183ef80e0ee0942013cc776e1be5c78da9f66e0"
Jul 28 02:13:43.682009 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.681985    2555 scope.go:117] "RemoveContainer" containerID="53ec6007b1a558540dba36ea25b0dd8907be5b208b563c4c60108e76d5fe6475"
Jul 28 02:13:43.689220 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.689205    2555 scope.go:117] "RemoveContainer" containerID="090d8580ff2e55e9a7b8116e30e5cc7949e0ed4ad8e86fa3dd7a9a6b6fdaa914"
Jul 28 02:13:43.695799 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.695779    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"]
Jul 28 02:13:43.696368 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.696353    2555 scope.go:117] "RemoveContainer" containerID="6db5417e6581e9e22467907de183ef80e0ee0942013cc776e1be5c78da9f66e0"
Jul 28 02:13:43.696691 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:13:43.696673    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6db5417e6581e9e22467907de183ef80e0ee0942013cc776e1be5c78da9f66e0\": container with ID starting with 6db5417e6581e9e22467907de183ef80e0ee0942013cc776e1be5c78da9f66e0 not found: ID does not exist" containerID="6db5417e6581e9e22467907de183ef80e0ee0942013cc776e1be5c78da9f66e0"
Jul 28 02:13:43.696755 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.696700    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db5417e6581e9e22467907de183ef80e0ee0942013cc776e1be5c78da9f66e0"} err="failed to get container status \"6db5417e6581e9e22467907de183ef80e0ee0942013cc776e1be5c78da9f66e0\": rpc error: code = NotFound desc = could not find container \"6db5417e6581e9e22467907de183ef80e0ee0942013cc776e1be5c78da9f66e0\": container with ID starting with 6db5417e6581e9e22467907de183ef80e0ee0942013cc776e1be5c78da9f66e0 not found: ID does not exist"
Jul 28 02:13:43.696755 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.696718    2555 scope.go:117] "RemoveContainer" containerID="53ec6007b1a558540dba36ea25b0dd8907be5b208b563c4c60108e76d5fe6475"
Jul 28 02:13:43.696958 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:13:43.696942    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53ec6007b1a558540dba36ea25b0dd8907be5b208b563c4c60108e76d5fe6475\": container with ID starting with 53ec6007b1a558540dba36ea25b0dd8907be5b208b563c4c60108e76d5fe6475 not found: ID does not exist" containerID="53ec6007b1a558540dba36ea25b0dd8907be5b208b563c4c60108e76d5fe6475"
Jul 28 02:13:43.696998 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.696964    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ec6007b1a558540dba36ea25b0dd8907be5b208b563c4c60108e76d5fe6475"} err="failed to get container status \"53ec6007b1a558540dba36ea25b0dd8907be5b208b563c4c60108e76d5fe6475\": rpc error: code = NotFound desc = could not find container \"53ec6007b1a558540dba36ea25b0dd8907be5b208b563c4c60108e76d5fe6475\": container with ID starting with 53ec6007b1a558540dba36ea25b0dd8907be5b208b563c4c60108e76d5fe6475 not found: ID does not exist"
Jul 28 02:13:43.696998 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.696979    2555 scope.go:117] "RemoveContainer" containerID="090d8580ff2e55e9a7b8116e30e5cc7949e0ed4ad8e86fa3dd7a9a6b6fdaa914"
Jul 28 02:13:43.697200 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:13:43.697184    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090d8580ff2e55e9a7b8116e30e5cc7949e0ed4ad8e86fa3dd7a9a6b6fdaa914\": container with ID starting with 090d8580ff2e55e9a7b8116e30e5cc7949e0ed4ad8e86fa3dd7a9a6b6fdaa914 not found: ID does not exist" containerID="090d8580ff2e55e9a7b8116e30e5cc7949e0ed4ad8e86fa3dd7a9a6b6fdaa914"
Jul 28 02:13:43.697239 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.697204    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090d8580ff2e55e9a7b8116e30e5cc7949e0ed4ad8e86fa3dd7a9a6b6fdaa914"} err="failed to get container status \"090d8580ff2e55e9a7b8116e30e5cc7949e0ed4ad8e86fa3dd7a9a6b6fdaa914\": rpc error: code = NotFound desc = could not find container \"090d8580ff2e55e9a7b8116e30e5cc7949e0ed4ad8e86fa3dd7a9a6b6fdaa914\": container with ID starting with 090d8580ff2e55e9a7b8116e30e5cc7949e0ed4ad8e86fa3dd7a9a6b6fdaa914 not found: ID does not exist"
Jul 28 02:13:43.699629 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:43.699606    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8pzxm"]
Jul 28 02:13:45.451610 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.451514    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" path="/var/lib/kubelet/pods/e1fd040f-ee0f-4bf6-b032-fb985659c042/volumes"
Jul 28 02:13:45.983698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.983665    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"]
Jul 28 02:13:45.983917 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.983905    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="kube-rbac-proxy"
Jul 28 02:13:45.983961 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.983919    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="kube-rbac-proxy"
Jul 28 02:13:45.983961 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.983927    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="storage-initializer"
Jul 28 02:13:45.983961 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.983933    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="storage-initializer"
Jul 28 02:13:45.983961 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.983944    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="kserve-container"
Jul 28 02:13:45.983961 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.983950    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="kserve-container"
Jul 28 02:13:45.984115 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.983988    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="kube-rbac-proxy"
Jul 28 02:13:45.984115 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.983996    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1fd040f-ee0f-4bf6-b032-fb985659c042" containerName="kserve-container"
Jul 28 02:13:45.988382 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.988361    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:45.990938 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.990918    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 28 02:13:45.991062 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.990943    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:13:45.991062 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.990962    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:13:45.991941 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.991923    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:13:45.992042 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.991944    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:13:45.992042 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.991925    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:13:45.992042 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.991962    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 28 02:13:45.998873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:45.998849    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"]
Jul 28 02:13:46.071420 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.071383    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-k8bjq\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:46.071420 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.071420    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-k8bjq\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:46.071697 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.071483    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-k8bjq\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:46.071697 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.071548    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-k8bjq\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:46.071697 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.071634    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4kjn\" (UniqueName: \"kubernetes.io/projected/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kube-api-access-h4kjn\") pod \"isvc-pmml-predictor-8674795f59-k8bjq\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:46.172305 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.172258    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4kjn\" (UniqueName: \"kubernetes.io/projected/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kube-api-access-h4kjn\") pod \"isvc-pmml-predictor-8674795f59-k8bjq\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:46.172305 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.172311    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-k8bjq\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:46.172610 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.172328    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-k8bjq\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:46.172610 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.172451    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-k8bjq\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:46.172610 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.172489    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-k8bjq\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:46.172781 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.172747    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-k8bjq\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:46.173063 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.173044    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-k8bjq\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:46.173136 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.173118    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-k8bjq\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:46.175130 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.175114    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-k8bjq\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:46.181771 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.181752    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4kjn\" (UniqueName: \"kubernetes.io/projected/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kube-api-access-h4kjn\") pod \"isvc-pmml-predictor-8674795f59-k8bjq\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:46.298059 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.298025    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:46.429150 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.429119    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"]
Jul 28 02:13:46.431839 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:13:46.431804    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc64bf9a_4072_4eca_8b8f_2b7877fcef7a.slice/crio-4b9caeae0cf0bd41dd94829399d75c2feaa2756cccdf67e921a1b4d0a331d5de WatchSource:0}: Error finding container 4b9caeae0cf0bd41dd94829399d75c2feaa2756cccdf67e921a1b4d0a331d5de: Status 404 returned error can't find the container with id 4b9caeae0cf0bd41dd94829399d75c2feaa2756cccdf67e921a1b4d0a331d5de
Jul 28 02:13:46.685308 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.685224    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" event={"ID":"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a","Type":"ContainerStarted","Data":"d61bcda7bedf4cb4e5c5992e01915624b1172f83d5b7daa1d35299e6081cc5f6"}
Jul 28 02:13:46.685308 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:46.685293    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" event={"ID":"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a","Type":"ContainerStarted","Data":"4b9caeae0cf0bd41dd94829399d75c2feaa2756cccdf67e921a1b4d0a331d5de"}
Jul 28 02:13:50.700008 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:50.699978    2555 generic.go:356] "Generic (PLEG): container finished" podID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerID="d61bcda7bedf4cb4e5c5992e01915624b1172f83d5b7daa1d35299e6081cc5f6" exitCode=0
Jul 28 02:13:50.700359 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:50.700028    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" event={"ID":"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a","Type":"ContainerDied","Data":"d61bcda7bedf4cb4e5c5992e01915624b1172f83d5b7daa1d35299e6081cc5f6"}
Jul 28 02:13:57.733646 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:57.733617    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" event={"ID":"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a","Type":"ContainerStarted","Data":"a7dc01aae67fb129982c28a28d41fed7d82d58815d3595f053b091bffdb3ecf8"}
Jul 28 02:13:58.739213 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:58.739175    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" event={"ID":"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a","Type":"ContainerStarted","Data":"434777d7d634b848312a1d468636223a926ced9248a6b7a58187d5ccdb5aa961"}
Jul 28 02:13:58.739610 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:58.739281    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:58.759591 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:58.759543    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" podStartSLOduration=3.211303971 podStartE2EDuration="13.759508469s" totalImagesPullingTime="6.966860413s" totalInitContainerRuntime="3.581344085s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:13:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:13:58.758253264 +0000 UTC m=+2885.767771709" watchObservedRunningTime="2026-07-28 02:13:58.759508469 +0000 UTC m=+2885.769026894"
Jul 28 02:13:59.743097 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:59.743062    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:13:59.744371 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:13:59.744342    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 02:14:00.746742 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:14:00.746697    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 02:14:05.751749 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:14:05.751721    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:14:05.752243 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:14:05.752216    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 02:14:15.753154 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:14:15.753113    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 02:14:25.752189 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:14:25.752152    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 02:14:35.752504 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:14:35.752463    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 02:14:45.752560 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:14:45.752507    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 02:14:55.752744 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:14:55.752700    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 02:15:05.752860 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:05.752821    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 02:15:15.752727 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:15.752647    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:15:16.851615 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:16.851578    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"]
Jul 28 02:15:16.852001 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:16.851800    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kserve-container" containerID="cri-o://a7dc01aae67fb129982c28a28d41fed7d82d58815d3595f053b091bffdb3ecf8" gracePeriod=30
Jul 28 02:15:16.852001 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:16.851834    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kube-rbac-proxy" containerID="cri-o://434777d7d634b848312a1d468636223a926ced9248a6b7a58187d5ccdb5aa961" gracePeriod=30
Jul 28 02:15:16.997696 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:16.997665    2555 generic.go:356] "Generic (PLEG): container finished" podID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerID="434777d7d634b848312a1d468636223a926ced9248a6b7a58187d5ccdb5aa961" exitCode=2
Jul 28 02:15:16.997823 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:16.997738    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" event={"ID":"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a","Type":"ContainerDied","Data":"434777d7d634b848312a1d468636223a926ced9248a6b7a58187d5ccdb5aa961"}
Jul 28 02:15:20.294283 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:20.294254    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:15:20.392202 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:20.392121    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4kjn\" (UniqueName: \"kubernetes.io/projected/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kube-api-access-h4kjn\") pod \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") "
Jul 28 02:15:20.392202 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:20.392163    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-cabundle-cert\") pod \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") "
Jul 28 02:15:20.392202 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:20.392188    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kube-rbac-proxy-sar-config\") pod \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") "
Jul 28 02:15:20.392463 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:20.392217    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kserve-provision-location\") pod \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") "
Jul 28 02:15:20.392463 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:20.392252    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-proxy-tls\") pod \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\" (UID: \"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a\") "
Jul 28 02:15:20.392601 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:20.392542    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" (UID: "bc64bf9a-4072-4eca-8b8f-2b7877fcef7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:15:20.392649 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:20.392596    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" (UID: "bc64bf9a-4072-4eca-8b8f-2b7877fcef7a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:15:20.392649 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:20.392598    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" (UID: "bc64bf9a-4072-4eca-8b8f-2b7877fcef7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:15:20.394231 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:20.394206    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kube-api-access-h4kjn" (OuterVolumeSpecName: "kube-api-access-h4kjn") pod "bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" (UID: "bc64bf9a-4072-4eca-8b8f-2b7877fcef7a"). InnerVolumeSpecName "kube-api-access-h4kjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:15:20.394324 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:20.394289    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" (UID: "bc64bf9a-4072-4eca-8b8f-2b7877fcef7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:15:20.493687 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:20.493661    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4kjn\" (UniqueName: \"kubernetes.io/projected/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kube-api-access-h4kjn\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:15:20.493687 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:20.493683    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:15:20.493828 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:20.493693    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:15:20.493828 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:20.493701    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:15:20.493828 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:20.493711    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:15:21.013035 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:21.012996    2555 generic.go:356] "Generic (PLEG): container finished" podID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerID="a7dc01aae67fb129982c28a28d41fed7d82d58815d3595f053b091bffdb3ecf8" exitCode=0
Jul 28 02:15:21.013210 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:21.013054    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"
Jul 28 02:15:21.013210 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:21.013084    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" event={"ID":"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a","Type":"ContainerDied","Data":"a7dc01aae67fb129982c28a28d41fed7d82d58815d3595f053b091bffdb3ecf8"}
Jul 28 02:15:21.013210 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:21.013141    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq" event={"ID":"bc64bf9a-4072-4eca-8b8f-2b7877fcef7a","Type":"ContainerDied","Data":"4b9caeae0cf0bd41dd94829399d75c2feaa2756cccdf67e921a1b4d0a331d5de"}
Jul 28 02:15:21.013210 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:21.013169    2555 scope.go:117] "RemoveContainer" containerID="434777d7d634b848312a1d468636223a926ced9248a6b7a58187d5ccdb5aa961"
Jul 28 02:15:21.021119 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:21.021105    2555 scope.go:117] "RemoveContainer" containerID="a7dc01aae67fb129982c28a28d41fed7d82d58815d3595f053b091bffdb3ecf8"
Jul 28 02:15:21.028394 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:21.028378    2555 scope.go:117] "RemoveContainer" containerID="d61bcda7bedf4cb4e5c5992e01915624b1172f83d5b7daa1d35299e6081cc5f6"
Jul 28 02:15:21.034979 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:21.034957    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"]
Jul 28 02:15:21.036017 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:21.036004    2555 scope.go:117] "RemoveContainer" containerID="434777d7d634b848312a1d468636223a926ced9248a6b7a58187d5ccdb5aa961"
Jul 28 02:15:21.036326 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:15:21.036305    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"434777d7d634b848312a1d468636223a926ced9248a6b7a58187d5ccdb5aa961\": container with ID starting with 434777d7d634b848312a1d468636223a926ced9248a6b7a58187d5ccdb5aa961 not found: ID does not exist" containerID="434777d7d634b848312a1d468636223a926ced9248a6b7a58187d5ccdb5aa961"
Jul 28 02:15:21.036403 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:21.036339    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"434777d7d634b848312a1d468636223a926ced9248a6b7a58187d5ccdb5aa961"} err="failed to get container status \"434777d7d634b848312a1d468636223a926ced9248a6b7a58187d5ccdb5aa961\": rpc error: code = NotFound desc = could not find container \"434777d7d634b848312a1d468636223a926ced9248a6b7a58187d5ccdb5aa961\": container with ID starting with 434777d7d634b848312a1d468636223a926ced9248a6b7a58187d5ccdb5aa961 not found: ID does not exist"
Jul 28 02:15:21.036403 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:21.036363    2555 scope.go:117] "RemoveContainer" containerID="a7dc01aae67fb129982c28a28d41fed7d82d58815d3595f053b091bffdb3ecf8"
Jul 28 02:15:21.036772 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:15:21.036706    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7dc01aae67fb129982c28a28d41fed7d82d58815d3595f053b091bffdb3ecf8\": container with ID starting with a7dc01aae67fb129982c28a28d41fed7d82d58815d3595f053b091bffdb3ecf8 not found: ID does not exist" containerID="a7dc01aae67fb129982c28a28d41fed7d82d58815d3595f053b091bffdb3ecf8"
Jul 28 02:15:21.036772 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:21.036743    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7dc01aae67fb129982c28a28d41fed7d82d58815d3595f053b091bffdb3ecf8"} err="failed to get container status \"a7dc01aae67fb129982c28a28d41fed7d82d58815d3595f053b091bffdb3ecf8\": rpc error: code = NotFound desc = could not find container \"a7dc01aae67fb129982c28a28d41fed7d82d58815d3595f053b091bffdb3ecf8\": container with ID starting with a7dc01aae67fb129982c28a28d41fed7d82d58815d3595f053b091bffdb3ecf8 not found: ID does not exist"
Jul 28 02:15:21.036772 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:21.036765    2555 scope.go:117] "RemoveContainer" containerID="d61bcda7bedf4cb4e5c5992e01915624b1172f83d5b7daa1d35299e6081cc5f6"
Jul 28 02:15:21.037357 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:15:21.037340    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61bcda7bedf4cb4e5c5992e01915624b1172f83d5b7daa1d35299e6081cc5f6\": container with ID starting with d61bcda7bedf4cb4e5c5992e01915624b1172f83d5b7daa1d35299e6081cc5f6 not found: ID does not exist" containerID="d61bcda7bedf4cb4e5c5992e01915624b1172f83d5b7daa1d35299e6081cc5f6"
Jul 28 02:15:21.037418 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:21.037364    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61bcda7bedf4cb4e5c5992e01915624b1172f83d5b7daa1d35299e6081cc5f6"} err="failed to get container status \"d61bcda7bedf4cb4e5c5992e01915624b1172f83d5b7daa1d35299e6081cc5f6\": rpc error: code = NotFound desc = could not find container \"d61bcda7bedf4cb4e5c5992e01915624b1172f83d5b7daa1d35299e6081cc5f6\": container with ID starting with d61bcda7bedf4cb4e5c5992e01915624b1172f83d5b7daa1d35299e6081cc5f6 not found: ID does not exist"
Jul 28 02:15:21.038101 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:21.038083    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-k8bjq"]
Jul 28 02:15:21.451219 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:21.451141    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" path="/var/lib/kubelet/pods/bc64bf9a-4072-4eca-8b8f-2b7877fcef7a/volumes"
Jul 28 02:15:22.015010 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.014982    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"]
Jul 28 02:15:22.015238 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.015226    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="storage-initializer"
Jul 28 02:15:22.015287 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.015240    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="storage-initializer"
Jul 28 02:15:22.015287 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.015252    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kserve-container"
Jul 28 02:15:22.015287 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.015258    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kserve-container"
Jul 28 02:15:22.015287 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.015270    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kube-rbac-proxy"
Jul 28 02:15:22.015287 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.015276    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kube-rbac-proxy"
Jul 28 02:15:22.015463 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.015336    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kserve-container"
Jul 28 02:15:22.015463 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.015343    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc64bf9a-4072-4eca-8b8f-2b7877fcef7a" containerName="kube-rbac-proxy"
Jul 28 02:15:22.018625 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.018605    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.022158 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.022134    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 28 02:15:22.022578 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.022560    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:15:22.022642 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.022580    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:15:22.022642 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.022560    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:15:22.023366 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.023348    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:15:22.023485 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.023375    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:15:22.023485 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.023432    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 28 02:15:22.028928 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.028903    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"]
Jul 28 02:15:22.106707 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.106685    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5f787ec-7b77-435a-bbff-9cd7691d2f53-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-cprhm\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.106843 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.106720    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dqb8\" (UniqueName: \"kubernetes.io/projected/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kube-api-access-5dqb8\") pod \"isvc-pmml-runtime-predictor-885fc8777-cprhm\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.106843 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.106757    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-cprhm\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.106843 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.106826    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5f787ec-7b77-435a-bbff-9cd7691d2f53-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-cprhm\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.107006 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.106865    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-cprhm\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.207740 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.207712    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dqb8\" (UniqueName: \"kubernetes.io/projected/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kube-api-access-5dqb8\") pod \"isvc-pmml-runtime-predictor-885fc8777-cprhm\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.207855 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.207765    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-cprhm\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.207900 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.207871    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5f787ec-7b77-435a-bbff-9cd7691d2f53-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-cprhm\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.207934 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.207902    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-cprhm\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.207934 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.207925    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5f787ec-7b77-435a-bbff-9cd7691d2f53-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-cprhm\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.208106 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.208090    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-cprhm\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.208463 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.208444    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5f787ec-7b77-435a-bbff-9cd7691d2f53-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-cprhm\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.208604 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.208586    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-cprhm\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.210409 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.210390    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5f787ec-7b77-435a-bbff-9cd7691d2f53-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-cprhm\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.216302 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.216282    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dqb8\" (UniqueName: \"kubernetes.io/projected/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kube-api-access-5dqb8\") pod \"isvc-pmml-runtime-predictor-885fc8777-cprhm\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.328724 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.328642    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:22.460191 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:22.460150    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"]
Jul 28 02:15:22.463198 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:15:22.463173    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f787ec_7b77_435a_bbff_9cd7691d2f53.slice/crio-5da20dfdb574a1dfac76a1f7319ce21535d67a047b31688533c8f40ef3e0a5c6 WatchSource:0}: Error finding container 5da20dfdb574a1dfac76a1f7319ce21535d67a047b31688533c8f40ef3e0a5c6: Status 404 returned error can't find the container with id 5da20dfdb574a1dfac76a1f7319ce21535d67a047b31688533c8f40ef3e0a5c6
Jul 28 02:15:23.022981 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:23.022943    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" event={"ID":"d5f787ec-7b77-435a-bbff-9cd7691d2f53","Type":"ContainerStarted","Data":"52110624b640a71cb21a0a01c11ee09a2f3cf7e40cc1f37710fc00cef29c584f"}
Jul 28 02:15:23.023165 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:23.023034    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" event={"ID":"d5f787ec-7b77-435a-bbff-9cd7691d2f53","Type":"ContainerStarted","Data":"5da20dfdb574a1dfac76a1f7319ce21535d67a047b31688533c8f40ef3e0a5c6"}
Jul 28 02:15:27.037648 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:27.037613    2555 generic.go:356] "Generic (PLEG): container finished" podID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerID="52110624b640a71cb21a0a01c11ee09a2f3cf7e40cc1f37710fc00cef29c584f" exitCode=0
Jul 28 02:15:27.038084 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:27.037689    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" event={"ID":"d5f787ec-7b77-435a-bbff-9cd7691d2f53","Type":"ContainerDied","Data":"52110624b640a71cb21a0a01c11ee09a2f3cf7e40cc1f37710fc00cef29c584f"}
Jul 28 02:15:28.042844 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:28.042810    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" event={"ID":"d5f787ec-7b77-435a-bbff-9cd7691d2f53","Type":"ContainerStarted","Data":"10e3dc3eab80fd8601ac82cab2be309a108a6381c385784746ff2cf5c174ae0b"}
Jul 28 02:15:28.043195 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:28.042876    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" event={"ID":"d5f787ec-7b77-435a-bbff-9cd7691d2f53","Type":"ContainerStarted","Data":"0a0f69ee9129a62d9629d157a0ee524be0e36ee2b24649d247a82033710fde42"}
Jul 28 02:15:28.043195 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:28.043154    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:28.043291 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:28.043201    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:28.044573 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:28.044546    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 02:15:28.062384 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:28.062342    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" podStartSLOduration=3.136138941 podStartE2EDuration="7.062329355s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.926190414s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:15:28.062106941 +0000 UTC m=+2975.071625384" watchObservedRunningTime="2026-07-28 02:15:28.062329355 +0000 UTC m=+2975.071847780"
Jul 28 02:15:29.046216 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:29.046179    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 02:15:34.050251 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:34.050220    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:15:34.050755 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:34.050730    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 02:15:44.051498 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:44.051463    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 02:15:54.051302 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:15:54.051253    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 02:16:04.050877 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:04.050836    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 02:16:14.050784 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:14.050740    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 02:16:24.050901 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:24.050859    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 02:16:34.050772 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:34.050736    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 02:16:44.051398 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:44.051364    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:16:53.081283 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:53.081251    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"]
Jul 28 02:16:53.081708 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:53.081503    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kserve-container" containerID="cri-o://0a0f69ee9129a62d9629d157a0ee524be0e36ee2b24649d247a82033710fde42" gracePeriod=30
Jul 28 02:16:53.081708 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:53.081581    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kube-rbac-proxy" containerID="cri-o://10e3dc3eab80fd8601ac82cab2be309a108a6381c385784746ff2cf5c174ae0b" gracePeriod=30
Jul 28 02:16:53.342643 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:53.342541    2555 generic.go:356] "Generic (PLEG): container finished" podID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerID="10e3dc3eab80fd8601ac82cab2be309a108a6381c385784746ff2cf5c174ae0b" exitCode=2
Jul 28 02:16:53.342643 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:53.342601    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" event={"ID":"d5f787ec-7b77-435a-bbff-9cd7691d2f53","Type":"ContainerDied","Data":"10e3dc3eab80fd8601ac82cab2be309a108a6381c385784746ff2cf5c174ae0b"}
Jul 28 02:16:54.046409 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:54.046367    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 28 02:16:54.050719 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:54.050687    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 02:16:56.725275 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:56.725251    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:16:56.777009 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:56.776970    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5f787ec-7b77-435a-bbff-9cd7691d2f53-cabundle-cert\") pod \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") "
Jul 28 02:16:56.777183 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:56.777030    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kube-rbac-proxy-sar-config\") pod \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") "
Jul 28 02:16:56.777183 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:56.777081    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dqb8\" (UniqueName: \"kubernetes.io/projected/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kube-api-access-5dqb8\") pod \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") "
Jul 28 02:16:56.777183 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:56.777115    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kserve-provision-location\") pod \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") "
Jul 28 02:16:56.777183 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:56.777141    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5f787ec-7b77-435a-bbff-9cd7691d2f53-proxy-tls\") pod \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\" (UID: \"d5f787ec-7b77-435a-bbff-9cd7691d2f53\") "
Jul 28 02:16:56.777439 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:56.777409    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f787ec-7b77-435a-bbff-9cd7691d2f53-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5f787ec-7b77-435a-bbff-9cd7691d2f53" (UID: "d5f787ec-7b77-435a-bbff-9cd7691d2f53"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:16:56.777439 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:56.777422    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5f787ec-7b77-435a-bbff-9cd7691d2f53" (UID: "d5f787ec-7b77-435a-bbff-9cd7691d2f53"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:16:56.777587 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:56.777459    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5f787ec-7b77-435a-bbff-9cd7691d2f53" (UID: "d5f787ec-7b77-435a-bbff-9cd7691d2f53"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:16:56.779340 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:56.779318    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kube-api-access-5dqb8" (OuterVolumeSpecName: "kube-api-access-5dqb8") pod "d5f787ec-7b77-435a-bbff-9cd7691d2f53" (UID: "d5f787ec-7b77-435a-bbff-9cd7691d2f53"). InnerVolumeSpecName "kube-api-access-5dqb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:16:56.779444 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:56.779382    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f787ec-7b77-435a-bbff-9cd7691d2f53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5f787ec-7b77-435a-bbff-9cd7691d2f53" (UID: "d5f787ec-7b77-435a-bbff-9cd7691d2f53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:16:56.878647 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:56.878551    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:16:56.878647 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:56.878579    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dqb8\" (UniqueName: \"kubernetes.io/projected/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kube-api-access-5dqb8\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:16:56.878647 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:56.878589    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5f787ec-7b77-435a-bbff-9cd7691d2f53-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:16:56.878647 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:56.878599    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5f787ec-7b77-435a-bbff-9cd7691d2f53-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:16:56.878647 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:56.878608    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5f787ec-7b77-435a-bbff-9cd7691d2f53-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:16:57.357849 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:57.357818    2555 generic.go:356] "Generic (PLEG): container finished" podID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerID="0a0f69ee9129a62d9629d157a0ee524be0e36ee2b24649d247a82033710fde42" exitCode=0
Jul 28 02:16:57.358019 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:57.357877    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"
Jul 28 02:16:57.358019 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:57.357885    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" event={"ID":"d5f787ec-7b77-435a-bbff-9cd7691d2f53","Type":"ContainerDied","Data":"0a0f69ee9129a62d9629d157a0ee524be0e36ee2b24649d247a82033710fde42"}
Jul 28 02:16:57.358019 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:57.357935    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm" event={"ID":"d5f787ec-7b77-435a-bbff-9cd7691d2f53","Type":"ContainerDied","Data":"5da20dfdb574a1dfac76a1f7319ce21535d67a047b31688533c8f40ef3e0a5c6"}
Jul 28 02:16:57.358019 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:57.357960    2555 scope.go:117] "RemoveContainer" containerID="10e3dc3eab80fd8601ac82cab2be309a108a6381c385784746ff2cf5c174ae0b"
Jul 28 02:16:57.366372 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:57.366355    2555 scope.go:117] "RemoveContainer" containerID="0a0f69ee9129a62d9629d157a0ee524be0e36ee2b24649d247a82033710fde42"
Jul 28 02:16:57.373556 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:57.373516    2555 scope.go:117] "RemoveContainer" containerID="52110624b640a71cb21a0a01c11ee09a2f3cf7e40cc1f37710fc00cef29c584f"
Jul 28 02:16:57.379165 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:57.379145    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"]
Jul 28 02:16:57.383124 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:57.383094    2555 scope.go:117] "RemoveContainer" containerID="10e3dc3eab80fd8601ac82cab2be309a108a6381c385784746ff2cf5c174ae0b"
Jul 28 02:16:57.383584 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:16:57.383562    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e3dc3eab80fd8601ac82cab2be309a108a6381c385784746ff2cf5c174ae0b\": container with ID starting with 10e3dc3eab80fd8601ac82cab2be309a108a6381c385784746ff2cf5c174ae0b not found: ID does not exist" containerID="10e3dc3eab80fd8601ac82cab2be309a108a6381c385784746ff2cf5c174ae0b"
Jul 28 02:16:57.383704 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:57.383591    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e3dc3eab80fd8601ac82cab2be309a108a6381c385784746ff2cf5c174ae0b"} err="failed to get container status \"10e3dc3eab80fd8601ac82cab2be309a108a6381c385784746ff2cf5c174ae0b\": rpc error: code = NotFound desc = could not find container \"10e3dc3eab80fd8601ac82cab2be309a108a6381c385784746ff2cf5c174ae0b\": container with ID starting with 10e3dc3eab80fd8601ac82cab2be309a108a6381c385784746ff2cf5c174ae0b not found: ID does not exist"
Jul 28 02:16:57.383704 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:57.383608    2555 scope.go:117] "RemoveContainer" containerID="0a0f69ee9129a62d9629d157a0ee524be0e36ee2b24649d247a82033710fde42"
Jul 28 02:16:57.383908 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:16:57.383890    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a0f69ee9129a62d9629d157a0ee524be0e36ee2b24649d247a82033710fde42\": container with ID starting with 0a0f69ee9129a62d9629d157a0ee524be0e36ee2b24649d247a82033710fde42 not found: ID does not exist" containerID="0a0f69ee9129a62d9629d157a0ee524be0e36ee2b24649d247a82033710fde42"
Jul 28 02:16:57.383968 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:57.383913    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a0f69ee9129a62d9629d157a0ee524be0e36ee2b24649d247a82033710fde42"} err="failed to get container status \"0a0f69ee9129a62d9629d157a0ee524be0e36ee2b24649d247a82033710fde42\": rpc error: code = NotFound desc = could not find container \"0a0f69ee9129a62d9629d157a0ee524be0e36ee2b24649d247a82033710fde42\": container with ID starting with 0a0f69ee9129a62d9629d157a0ee524be0e36ee2b24649d247a82033710fde42 not found: ID does not exist"
Jul 28 02:16:57.383968 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:57.383929    2555 scope.go:117] "RemoveContainer" containerID="52110624b640a71cb21a0a01c11ee09a2f3cf7e40cc1f37710fc00cef29c584f"
Jul 28 02:16:57.384210 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:16:57.384187    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52110624b640a71cb21a0a01c11ee09a2f3cf7e40cc1f37710fc00cef29c584f\": container with ID starting with 52110624b640a71cb21a0a01c11ee09a2f3cf7e40cc1f37710fc00cef29c584f not found: ID does not exist" containerID="52110624b640a71cb21a0a01c11ee09a2f3cf7e40cc1f37710fc00cef29c584f"
Jul 28 02:16:57.384284 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:57.384218    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52110624b640a71cb21a0a01c11ee09a2f3cf7e40cc1f37710fc00cef29c584f"} err="failed to get container status \"52110624b640a71cb21a0a01c11ee09a2f3cf7e40cc1f37710fc00cef29c584f\": rpc error: code = NotFound desc = could not find container \"52110624b640a71cb21a0a01c11ee09a2f3cf7e40cc1f37710fc00cef29c584f\": container with ID starting with 52110624b640a71cb21a0a01c11ee09a2f3cf7e40cc1f37710fc00cef29c584f not found: ID does not exist"
Jul 28 02:16:57.384697 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:57.384677    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-cprhm"]
Jul 28 02:16:57.451861 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:57.451827    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" path="/var/lib/kubelet/pods/d5f787ec-7b77-435a-bbff-9cd7691d2f53/volumes"
Jul 28 02:16:58.202556 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.202507    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"]
Jul 28 02:16:58.202945 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.202795    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="storage-initializer"
Jul 28 02:16:58.202945 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.202806    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="storage-initializer"
Jul 28 02:16:58.202945 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.202814    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kserve-container"
Jul 28 02:16:58.202945 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.202821    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kserve-container"
Jul 28 02:16:58.202945 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.202827    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kube-rbac-proxy"
Jul 28 02:16:58.202945 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.202833    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kube-rbac-proxy"
Jul 28 02:16:58.202945 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.202890    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kserve-container"
Jul 28 02:16:58.202945 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.202897    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f787ec-7b77-435a-bbff-9cd7691d2f53" containerName="kube-rbac-proxy"
Jul 28 02:16:58.207264 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.207247    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:58.209810 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.209787    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 28 02:16:58.209939 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.209848    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 02:16:58.210009 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.209934    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:16:58.210365 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.210351    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:16:58.210921 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.210902    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:16:58.210921 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.210912    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:16:58.211054 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.210912    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:16:58.217812 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.217791    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"]
Jul 28 02:16:58.290099 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.290060    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf2zc\" (UniqueName: \"kubernetes.io/projected/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kube-api-access-xf2zc\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:58.290099 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.290103    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:58.290325 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.290125    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:58.290325 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.290141    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:58.290325 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.290206    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:58.391392 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.391351    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf2zc\" (UniqueName: \"kubernetes.io/projected/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kube-api-access-xf2zc\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:58.391392 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.391397    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:58.391649 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.391419    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:58.391649 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.391486    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:58.391649 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:16:58.391505    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Jul 28 02:16:58.391649 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.391571    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:58.391649 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:16:58.391613    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-proxy-tls podName:e5fbb619-90ba-489c-a40b-f4d01c45cd3b nodeName:}" failed. No retries permitted until 2026-07-28 02:16:58.891591321 +0000 UTC m=+3065.901109723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" (UID: "e5fbb619-90ba-489c-a40b-f4d01c45cd3b") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Jul 28 02:16:58.391881 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.391856    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:58.392323 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.392301    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:58.392403 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.392337    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:58.402808 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.402786    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf2zc\" (UniqueName: \"kubernetes.io/projected/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kube-api-access-xf2zc\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:58.895678 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.895632    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:58.898175 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:58.898154    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:59.118165 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:59.118114    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:16:59.245283 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:59.245249    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"]
Jul 28 02:16:59.248171 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:16:59.248143    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5fbb619_90ba_489c_a40b_f4d01c45cd3b.slice/crio-6ff1ab16bdc4740e6a3f1295a8c5096447497ab2befe05d83970f1ed8484577d WatchSource:0}: Error finding container 6ff1ab16bdc4740e6a3f1295a8c5096447497ab2befe05d83970f1ed8484577d: Status 404 returned error can't find the container with id 6ff1ab16bdc4740e6a3f1295a8c5096447497ab2befe05d83970f1ed8484577d
Jul 28 02:16:59.366380 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:59.366345    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" event={"ID":"e5fbb619-90ba-489c-a40b-f4d01c45cd3b","Type":"ContainerStarted","Data":"8a8bfd045cb999ce0a91f8654fcb0d44a91c4cf5fe53539d0726d3dbdd8030e8"}
Jul 28 02:16:59.366600 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:16:59.366417    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" event={"ID":"e5fbb619-90ba-489c-a40b-f4d01c45cd3b","Type":"ContainerStarted","Data":"6ff1ab16bdc4740e6a3f1295a8c5096447497ab2befe05d83970f1ed8484577d"}
Jul 28 02:17:03.381491 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:17:03.381462    2555 generic.go:356] "Generic (PLEG): container finished" podID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerID="8a8bfd045cb999ce0a91f8654fcb0d44a91c4cf5fe53539d0726d3dbdd8030e8" exitCode=0
Jul 28 02:17:03.381937 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:17:03.381562    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" event={"ID":"e5fbb619-90ba-489c-a40b-f4d01c45cd3b","Type":"ContainerDied","Data":"8a8bfd045cb999ce0a91f8654fcb0d44a91c4cf5fe53539d0726d3dbdd8030e8"}
Jul 28 02:17:04.387215 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:17:04.387178    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" event={"ID":"e5fbb619-90ba-489c-a40b-f4d01c45cd3b","Type":"ContainerStarted","Data":"fe4d2fda3061a136632947ca8915fa96d8c495d8941926da39be998dfa935a3f"}
Jul 28 02:17:04.387633 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:17:04.387247    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" event={"ID":"e5fbb619-90ba-489c-a40b-f4d01c45cd3b","Type":"ContainerStarted","Data":"d63a0710c1a99c4450f37bd0b4dba983079ec13cf00c91a152f1d3a7172dcedd"}
Jul 28 02:17:04.387633 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:17:04.387445    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:17:04.409706 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:17:04.409649    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" podStartSLOduration=2.783907849 podStartE2EDuration="6.409632506s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.625724657s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:17:04.408841734 +0000 UTC m=+3071.418360162" watchObservedRunningTime="2026-07-28 02:17:04.409632506 +0000 UTC m=+3071.419150931"
Jul 28 02:17:05.390831 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:17:05.390796    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:17:05.392084 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:17:05.392057    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 02:17:06.394876 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:17:06.394829    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 02:17:11.399283 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:17:11.399254    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:17:11.399964 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:17:11.399934    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 02:17:21.400086 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:17:21.400044    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 02:17:31.400005 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:17:31.399964    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 02:17:41.400745 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:17:41.400702    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 02:17:51.400636 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:17:51.400600    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 02:18:01.399869 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:01.399832    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 02:18:11.400827 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:11.400778    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 02:18:21.400782 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:21.400713    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:18:29.551888 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:29.551847    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"]
Jul 28 02:18:29.552352 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:29.552138    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kserve-container" containerID="cri-o://d63a0710c1a99c4450f37bd0b4dba983079ec13cf00c91a152f1d3a7172dcedd" gracePeriod=30
Jul 28 02:18:29.552352 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:29.552168    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kube-rbac-proxy" containerID="cri-o://fe4d2fda3061a136632947ca8915fa96d8c495d8941926da39be998dfa935a3f" gracePeriod=30
Jul 28 02:18:29.677849 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:29.677824    2555 generic.go:356] "Generic (PLEG): container finished" podID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerID="fe4d2fda3061a136632947ca8915fa96d8c495d8941926da39be998dfa935a3f" exitCode=2
Jul 28 02:18:29.677956 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:29.677880    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" event={"ID":"e5fbb619-90ba-489c-a40b-f4d01c45cd3b","Type":"ContainerDied","Data":"fe4d2fda3061a136632947ca8915fa96d8c495d8941926da39be998dfa935a3f"}
Jul 28 02:18:31.395868 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:31.395819    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Jul 28 02:18:31.400179 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:31.400145    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 02:18:32.693704 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:32.691498    2555 generic.go:356] "Generic (PLEG): container finished" podID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerID="d63a0710c1a99c4450f37bd0b4dba983079ec13cf00c91a152f1d3a7172dcedd" exitCode=0
Jul 28 02:18:32.693704 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:32.691612    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" event={"ID":"e5fbb619-90ba-489c-a40b-f4d01c45cd3b","Type":"ContainerDied","Data":"d63a0710c1a99c4450f37bd0b4dba983079ec13cf00c91a152f1d3a7172dcedd"}
Jul 28 02:18:32.792768 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:32.792744    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:18:32.949448 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:32.949360    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kube-rbac-proxy-sar-config\") pod \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") "
Jul 28 02:18:32.949448 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:32.949411    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kserve-provision-location\") pod \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") "
Jul 28 02:18:32.949448 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:32.949442    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf2zc\" (UniqueName: \"kubernetes.io/projected/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kube-api-access-xf2zc\") pod \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") "
Jul 28 02:18:32.949714 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:32.949472    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-cabundle-cert\") pod \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") "
Jul 28 02:18:32.949714 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:32.949493    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-proxy-tls\") pod \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\" (UID: \"e5fbb619-90ba-489c-a40b-f4d01c45cd3b\") "
Jul 28 02:18:32.949876 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:32.949843    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e5fbb619-90ba-489c-a40b-f4d01c45cd3b" (UID: "e5fbb619-90ba-489c-a40b-f4d01c45cd3b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:18:32.949998 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:32.949877    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e5fbb619-90ba-489c-a40b-f4d01c45cd3b" (UID: "e5fbb619-90ba-489c-a40b-f4d01c45cd3b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:18:32.949998 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:32.949848    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e5fbb619-90ba-489c-a40b-f4d01c45cd3b" (UID: "e5fbb619-90ba-489c-a40b-f4d01c45cd3b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:18:32.951709 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:32.951679    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kube-api-access-xf2zc" (OuterVolumeSpecName: "kube-api-access-xf2zc") pod "e5fbb619-90ba-489c-a40b-f4d01c45cd3b" (UID: "e5fbb619-90ba-489c-a40b-f4d01c45cd3b"). InnerVolumeSpecName "kube-api-access-xf2zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:18:32.951794 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:32.951706    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e5fbb619-90ba-489c-a40b-f4d01c45cd3b" (UID: "e5fbb619-90ba-489c-a40b-f4d01c45cd3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:18:33.050893 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:33.050865    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:18:33.050893 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:33.050891    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:18:33.051050 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:33.050900    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:18:33.051050 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:33.050909    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:18:33.051050 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:33.050918    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf2zc\" (UniqueName: \"kubernetes.io/projected/e5fbb619-90ba-489c-a40b-f4d01c45cd3b-kube-api-access-xf2zc\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:18:33.696773 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:33.696746    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"
Jul 28 02:18:33.696773 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:33.696746    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g" event={"ID":"e5fbb619-90ba-489c-a40b-f4d01c45cd3b","Type":"ContainerDied","Data":"6ff1ab16bdc4740e6a3f1295a8c5096447497ab2befe05d83970f1ed8484577d"}
Jul 28 02:18:33.697241 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:33.696818    2555 scope.go:117] "RemoveContainer" containerID="fe4d2fda3061a136632947ca8915fa96d8c495d8941926da39be998dfa935a3f"
Jul 28 02:18:33.705055 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:33.705032    2555 scope.go:117] "RemoveContainer" containerID="d63a0710c1a99c4450f37bd0b4dba983079ec13cf00c91a152f1d3a7172dcedd"
Jul 28 02:18:33.712006 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:33.711987    2555 scope.go:117] "RemoveContainer" containerID="8a8bfd045cb999ce0a91f8654fcb0d44a91c4cf5fe53539d0726d3dbdd8030e8"
Jul 28 02:18:33.713849 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:33.713828    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"]
Jul 28 02:18:33.717871 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:33.717838    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bvq9g"]
Jul 28 02:18:34.696841 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.696804    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"]
Jul 28 02:18:34.697291 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.697156    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kube-rbac-proxy"
Jul 28 02:18:34.697291 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.697173    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kube-rbac-proxy"
Jul 28 02:18:34.697291 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.697199    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="storage-initializer"
Jul 28 02:18:34.697291 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.697208    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="storage-initializer"
Jul 28 02:18:34.697291 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.697216    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kserve-container"
Jul 28 02:18:34.697291 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.697225    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kserve-container"
Jul 28 02:18:34.697291 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.697282    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kube-rbac-proxy"
Jul 28 02:18:34.697291 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.697293    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" containerName="kserve-container"
Jul 28 02:18:34.702046 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.702023    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:34.704738 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.704716    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:18:34.704824 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.704721    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a0f617-predictor-serving-cert\""
Jul 28 02:18:34.704990 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.704977    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:18:34.705076 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.705010    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:18:34.705752 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.705735    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:18:34.705752 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.705735    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a0f617-kube-rbac-proxy-sar-config\""
Jul 28 02:18:34.705889 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.705808    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:18:34.710208 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.710184    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"]
Jul 28 02:18:34.863489 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.863457    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a11cf713-5373-4a0a-9813-f1136bb9b315-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a0f617-predictor-5758d4969-69kf4\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") " pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:34.863629 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.863497    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnwdr\" (UniqueName: \"kubernetes.io/projected/a11cf713-5373-4a0a-9813-f1136bb9b315-kube-api-access-nnwdr\") pod \"isvc-primary-a0f617-predictor-5758d4969-69kf4\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") " pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:34.863629 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.863541    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a11cf713-5373-4a0a-9813-f1136bb9b315-proxy-tls\") pod \"isvc-primary-a0f617-predictor-5758d4969-69kf4\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") " pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:34.863629 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.863600    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a11cf713-5373-4a0a-9813-f1136bb9b315-kserve-provision-location\") pod \"isvc-primary-a0f617-predictor-5758d4969-69kf4\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") " pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:34.863740 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.863636    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a11cf713-5373-4a0a-9813-f1136bb9b315-cabundle-cert\") pod \"isvc-primary-a0f617-predictor-5758d4969-69kf4\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") " pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:34.964079 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.964001    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a11cf713-5373-4a0a-9813-f1136bb9b315-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a0f617-predictor-5758d4969-69kf4\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") " pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:34.964079 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.964040    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnwdr\" (UniqueName: \"kubernetes.io/projected/a11cf713-5373-4a0a-9813-f1136bb9b315-kube-api-access-nnwdr\") pod \"isvc-primary-a0f617-predictor-5758d4969-69kf4\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") " pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:34.964079 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.964061    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a11cf713-5373-4a0a-9813-f1136bb9b315-proxy-tls\") pod \"isvc-primary-a0f617-predictor-5758d4969-69kf4\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") " pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:34.964079 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.964081    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a11cf713-5373-4a0a-9813-f1136bb9b315-kserve-provision-location\") pod \"isvc-primary-a0f617-predictor-5758d4969-69kf4\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") " pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:34.964376 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.964098    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a11cf713-5373-4a0a-9813-f1136bb9b315-cabundle-cert\") pod \"isvc-primary-a0f617-predictor-5758d4969-69kf4\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") " pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:34.964605 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.964585    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a11cf713-5373-4a0a-9813-f1136bb9b315-kserve-provision-location\") pod \"isvc-primary-a0f617-predictor-5758d4969-69kf4\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") " pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:34.964849 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.964830    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a11cf713-5373-4a0a-9813-f1136bb9b315-cabundle-cert\") pod \"isvc-primary-a0f617-predictor-5758d4969-69kf4\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") " pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:34.964849 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.964843    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a11cf713-5373-4a0a-9813-f1136bb9b315-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a0f617-predictor-5758d4969-69kf4\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") " pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:34.966663 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.966645    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a11cf713-5373-4a0a-9813-f1136bb9b315-proxy-tls\") pod \"isvc-primary-a0f617-predictor-5758d4969-69kf4\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") " pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:34.973169 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:34.973146    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnwdr\" (UniqueName: \"kubernetes.io/projected/a11cf713-5373-4a0a-9813-f1136bb9b315-kube-api-access-nnwdr\") pod \"isvc-primary-a0f617-predictor-5758d4969-69kf4\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") " pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:35.012660 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:35.012630    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:35.143536 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:35.143485    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"]
Jul 28 02:18:35.145658 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:18:35.145625    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11cf713_5373_4a0a_9813_f1136bb9b315.slice/crio-7e2cd88bc8498bb46ce0e9b14d7d0afdf40fa78b765d4a2fb580e6651e5231db WatchSource:0}: Error finding container 7e2cd88bc8498bb46ce0e9b14d7d0afdf40fa78b765d4a2fb580e6651e5231db: Status 404 returned error can't find the container with id 7e2cd88bc8498bb46ce0e9b14d7d0afdf40fa78b765d4a2fb580e6651e5231db
Jul 28 02:18:35.452276 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:35.452247    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5fbb619-90ba-489c-a40b-f4d01c45cd3b" path="/var/lib/kubelet/pods/e5fbb619-90ba-489c-a40b-f4d01c45cd3b/volumes"
Jul 28 02:18:35.705917 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:35.705833    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" event={"ID":"a11cf713-5373-4a0a-9813-f1136bb9b315","Type":"ContainerStarted","Data":"2892014f8416db475fe79ed8e3b80a81ae0a2c30c8c28ce7c1d716a684bf557e"}
Jul 28 02:18:35.705917 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:35.705896    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" event={"ID":"a11cf713-5373-4a0a-9813-f1136bb9b315","Type":"ContainerStarted","Data":"7e2cd88bc8498bb46ce0e9b14d7d0afdf40fa78b765d4a2fb580e6651e5231db"}
Jul 28 02:18:39.720279 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:39.720241    2555 generic.go:356] "Generic (PLEG): container finished" podID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerID="2892014f8416db475fe79ed8e3b80a81ae0a2c30c8c28ce7c1d716a684bf557e" exitCode=0
Jul 28 02:18:39.720685 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:39.720315    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" event={"ID":"a11cf713-5373-4a0a-9813-f1136bb9b315","Type":"ContainerDied","Data":"2892014f8416db475fe79ed8e3b80a81ae0a2c30c8c28ce7c1d716a684bf557e"}
Jul 28 02:18:40.725319 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:40.725288    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" event={"ID":"a11cf713-5373-4a0a-9813-f1136bb9b315","Type":"ContainerStarted","Data":"7cc67881cfe13ee22d495b0ea4398afe3f5fe39ceda2e46e5df3d8397bbcf10a"}
Jul 28 02:18:40.725791 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:40.725355    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" event={"ID":"a11cf713-5373-4a0a-9813-f1136bb9b315","Type":"ContainerStarted","Data":"475debacea1eac000dec7caa8eb4f35f0f876e7313e0fe6490276b855a63c6b0"}
Jul 28 02:18:40.725791 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:40.725700    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:40.725791 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:40.725745    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:40.726975 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:40.726952    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 02:18:40.747622 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:40.747580    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" podStartSLOduration=3.228720449 podStartE2EDuration="6.74756671s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.518846261s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:18:40.747229098 +0000 UTC m=+3167.756747523" watchObservedRunningTime="2026-07-28 02:18:40.74756671 +0000 UTC m=+3167.757085135"
Jul 28 02:18:41.728969 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:41.728926    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 02:18:46.734876 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:46.734846    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:18:46.735380 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:46.735329    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 02:18:56.735747 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:18:56.735707    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 02:19:06.736161 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:19:06.736118    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 02:19:16.735482 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:19:16.735441    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 02:19:26.735414 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:19:26.735370    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 02:19:36.735497 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:19:36.735456    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 02:19:46.735926 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:19:46.735843    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 02:19:56.736238 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:19:56.736204    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:20:19.917105 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:19.917074    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"]
Jul 28 02:20:19.920390 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:19.920374    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:19.924273 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:19.924246    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a0f617-kube-rbac-proxy-sar-config\""
Jul 28 02:20:19.924273 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:19.924266    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-a0f617-dockercfg-b4k7j\""
Jul 28 02:20:19.924437 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:19.924256    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a0f617-predictor-serving-cert\""
Jul 28 02:20:19.924437 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:19.924248    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-a0f617\""
Jul 28 02:20:19.935101 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:19.935077    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"]
Jul 28 02:20:20.099316 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.095341    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kserve-provision-location\") pod \"isvc-secondary-a0f617-predictor-69747d6786-zcw7n\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") " pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:20.099316 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.095397    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzvrz\" (UniqueName: \"kubernetes.io/projected/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kube-api-access-pzvrz\") pod \"isvc-secondary-a0f617-predictor-69747d6786-zcw7n\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") " pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:20.099316 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.095430    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-cabundle-cert\") pod \"isvc-secondary-a0f617-predictor-69747d6786-zcw7n\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") " pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:20.099316 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.095472    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-proxy-tls\") pod \"isvc-secondary-a0f617-predictor-69747d6786-zcw7n\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") " pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:20.099316 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.095515    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a0f617-predictor-69747d6786-zcw7n\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") " pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:20.195952 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.195867    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kserve-provision-location\") pod \"isvc-secondary-a0f617-predictor-69747d6786-zcw7n\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") " pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:20.195952 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.195902    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvrz\" (UniqueName: \"kubernetes.io/projected/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kube-api-access-pzvrz\") pod \"isvc-secondary-a0f617-predictor-69747d6786-zcw7n\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") " pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:20.195952 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.195918    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-cabundle-cert\") pod \"isvc-secondary-a0f617-predictor-69747d6786-zcw7n\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") " pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:20.195952 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.195948    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-proxy-tls\") pod \"isvc-secondary-a0f617-predictor-69747d6786-zcw7n\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") " pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:20.196286 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.195982    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a0f617-predictor-69747d6786-zcw7n\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") " pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:20.196350 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.196316    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kserve-provision-location\") pod \"isvc-secondary-a0f617-predictor-69747d6786-zcw7n\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") " pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:20.196684 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.196655    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-cabundle-cert\") pod \"isvc-secondary-a0f617-predictor-69747d6786-zcw7n\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") " pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:20.196970 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.196766    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a0f617-predictor-69747d6786-zcw7n\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") " pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:20.198583 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.198554    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-proxy-tls\") pod \"isvc-secondary-a0f617-predictor-69747d6786-zcw7n\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") " pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:20.204920 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.204902    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzvrz\" (UniqueName: \"kubernetes.io/projected/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kube-api-access-pzvrz\") pod \"isvc-secondary-a0f617-predictor-69747d6786-zcw7n\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") " pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:20.230782 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.230754    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:20.363675 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:20.363647    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"]
Jul 28 02:20:20.365630 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:20:20.365598    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e31291_62c6_4f66_af91_c2dc5f2e0a03.slice/crio-8fce78e8723227b34d24d774a5d6cff9947413c4daf0c5aefe3b4926e6b5b1bd WatchSource:0}: Error finding container 8fce78e8723227b34d24d774a5d6cff9947413c4daf0c5aefe3b4926e6b5b1bd: Status 404 returned error can't find the container with id 8fce78e8723227b34d24d774a5d6cff9947413c4daf0c5aefe3b4926e6b5b1bd
Jul 28 02:20:21.064798 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:21.064755    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n" event={"ID":"f9e31291-62c6-4f66-af91-c2dc5f2e0a03","Type":"ContainerStarted","Data":"48a2c2d1acade670c5c731f3f5355296bc18c41a08a9660bf91338108700aa83"}
Jul 28 02:20:21.065207 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:21.064833    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n" event={"ID":"f9e31291-62c6-4f66-af91-c2dc5f2e0a03","Type":"ContainerStarted","Data":"8fce78e8723227b34d24d774a5d6cff9947413c4daf0c5aefe3b4926e6b5b1bd"}
Jul 28 02:20:25.080219 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:25.080188    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a0f617-predictor-69747d6786-zcw7n_f9e31291-62c6-4f66-af91-c2dc5f2e0a03/storage-initializer/0.log"
Jul 28 02:20:25.080721 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:25.080232    2555 generic.go:356] "Generic (PLEG): container finished" podID="f9e31291-62c6-4f66-af91-c2dc5f2e0a03" containerID="48a2c2d1acade670c5c731f3f5355296bc18c41a08a9660bf91338108700aa83" exitCode=1
Jul 28 02:20:25.080721 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:25.080282    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n" event={"ID":"f9e31291-62c6-4f66-af91-c2dc5f2e0a03","Type":"ContainerDied","Data":"48a2c2d1acade670c5c731f3f5355296bc18c41a08a9660bf91338108700aa83"}
Jul 28 02:20:26.085860 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:26.085830    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a0f617-predictor-69747d6786-zcw7n_f9e31291-62c6-4f66-af91-c2dc5f2e0a03/storage-initializer/0.log"
Jul 28 02:20:26.086227 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:26.085942    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n" event={"ID":"f9e31291-62c6-4f66-af91-c2dc5f2e0a03","Type":"ContainerStarted","Data":"a524784876b8b24c92b404df6559a5bf0feeb768b1f40ba58adc221efbca8837"}
Jul 28 02:20:27.091673 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:27.091645    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a0f617-predictor-69747d6786-zcw7n_f9e31291-62c6-4f66-af91-c2dc5f2e0a03/storage-initializer/1.log"
Jul 28 02:20:27.092107 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:27.092045    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a0f617-predictor-69747d6786-zcw7n_f9e31291-62c6-4f66-af91-c2dc5f2e0a03/storage-initializer/0.log"
Jul 28 02:20:27.092107 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:27.092079    2555 generic.go:356] "Generic (PLEG): container finished" podID="f9e31291-62c6-4f66-af91-c2dc5f2e0a03" containerID="a524784876b8b24c92b404df6559a5bf0feeb768b1f40ba58adc221efbca8837" exitCode=1
Jul 28 02:20:27.092180 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:27.092145    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n" event={"ID":"f9e31291-62c6-4f66-af91-c2dc5f2e0a03","Type":"ContainerDied","Data":"a524784876b8b24c92b404df6559a5bf0feeb768b1f40ba58adc221efbca8837"}
Jul 28 02:20:27.092219 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:27.092199    2555 scope.go:117] "RemoveContainer" containerID="48a2c2d1acade670c5c731f3f5355296bc18c41a08a9660bf91338108700aa83"
Jul 28 02:20:27.092579 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:27.092557    2555 scope.go:117] "RemoveContainer" containerID="48a2c2d1acade670c5c731f3f5355296bc18c41a08a9660bf91338108700aa83"
Jul 28 02:20:27.102962 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:20:27.102937    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-a0f617-predictor-69747d6786-zcw7n_kserve-ci-e2e-test(f9e31291-62c6-4f66-af91-c2dc5f2e0a03)\"" pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n" podUID="f9e31291-62c6-4f66-af91-c2dc5f2e0a03"
Jul 28 02:20:27.103413 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:20:27.103387    2555 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a0f617-predictor-69747d6786-zcw7n_kserve-ci-e2e-test_f9e31291-62c6-4f66-af91-c2dc5f2e0a03_0 in pod sandbox 8fce78e8723227b34d24d774a5d6cff9947413c4daf0c5aefe3b4926e6b5b1bd from index: no such id: '48a2c2d1acade670c5c731f3f5355296bc18c41a08a9660bf91338108700aa83'" containerID="48a2c2d1acade670c5c731f3f5355296bc18c41a08a9660bf91338108700aa83"
Jul 28 02:20:27.103495 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:27.103424    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a2c2d1acade670c5c731f3f5355296bc18c41a08a9660bf91338108700aa83"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a0f617-predictor-69747d6786-zcw7n_kserve-ci-e2e-test_f9e31291-62c6-4f66-af91-c2dc5f2e0a03_0 in pod sandbox 8fce78e8723227b34d24d774a5d6cff9947413c4daf0c5aefe3b4926e6b5b1bd from index: no such id: '48a2c2d1acade670c5c731f3f5355296bc18c41a08a9660bf91338108700aa83'"
Jul 28 02:20:28.097615 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:28.097582    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a0f617-predictor-69747d6786-zcw7n_f9e31291-62c6-4f66-af91-c2dc5f2e0a03/storage-initializer/1.log"
Jul 28 02:20:35.956024 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:35.955984    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"]
Jul 28 02:20:36.014455 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.014420    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"]
Jul 28 02:20:36.015144 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.014725    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kserve-container" containerID="cri-o://475debacea1eac000dec7caa8eb4f35f0f876e7313e0fe6490276b855a63c6b0" gracePeriod=30
Jul 28 02:20:36.015144 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.014764    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kube-rbac-proxy" containerID="cri-o://7cc67881cfe13ee22d495b0ea4398afe3f5fe39ceda2e46e5df3d8397bbcf10a" gracePeriod=30
Jul 28 02:20:36.127014 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.126991    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a0f617-predictor-69747d6786-zcw7n_f9e31291-62c6-4f66-af91-c2dc5f2e0a03/storage-initializer/1.log"
Jul 28 02:20:36.127127 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.127085    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n" event={"ID":"f9e31291-62c6-4f66-af91-c2dc5f2e0a03","Type":"ContainerDied","Data":"8fce78e8723227b34d24d774a5d6cff9947413c4daf0c5aefe3b4926e6b5b1bd"}
Jul 28 02:20:36.127185 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.127143    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fce78e8723227b34d24d774a5d6cff9947413c4daf0c5aefe3b4926e6b5b1bd"
Jul 28 02:20:36.133600 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.133581    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a0f617-predictor-69747d6786-zcw7n_f9e31291-62c6-4f66-af91-c2dc5f2e0a03/storage-initializer/1.log"
Jul 28 02:20:36.135725 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.135710    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:36.212265 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.212184    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-proxy-tls\") pod \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") "
Jul 28 02:20:36.212265 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.212248    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kserve-provision-location\") pod \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") "
Jul 28 02:20:36.212463 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.212283    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-cabundle-cert\") pod \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") "
Jul 28 02:20:36.212463 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.212307    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzvrz\" (UniqueName: \"kubernetes.io/projected/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kube-api-access-pzvrz\") pod \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") "
Jul 28 02:20:36.212463 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.212334    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kube-rbac-proxy-sar-config\") pod \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\" (UID: \"f9e31291-62c6-4f66-af91-c2dc5f2e0a03\") "
Jul 28 02:20:36.212629 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.212553    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9e31291-62c6-4f66-af91-c2dc5f2e0a03" (UID: "f9e31291-62c6-4f66-af91-c2dc5f2e0a03"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:20:36.212700 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.212677    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f9e31291-62c6-4f66-af91-c2dc5f2e0a03" (UID: "f9e31291-62c6-4f66-af91-c2dc5f2e0a03"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:20:36.212740 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.212710    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f9e31291-62c6-4f66-af91-c2dc5f2e0a03" (UID: "f9e31291-62c6-4f66-af91-c2dc5f2e0a03"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:20:36.214418 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.214395    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9e31291-62c6-4f66-af91-c2dc5f2e0a03" (UID: "f9e31291-62c6-4f66-af91-c2dc5f2e0a03"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:20:36.214571 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.214549    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kube-api-access-pzvrz" (OuterVolumeSpecName: "kube-api-access-pzvrz") pod "f9e31291-62c6-4f66-af91-c2dc5f2e0a03" (UID: "f9e31291-62c6-4f66-af91-c2dc5f2e0a03"). InnerVolumeSpecName "kube-api-access-pzvrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:20:36.312976 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.312940    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:20:36.312976 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.312970    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:20:36.312976 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.312979    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:20:36.312976 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.312987    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzvrz\" (UniqueName: \"kubernetes.io/projected/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kube-api-access-pzvrz\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:20:36.313226 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.312996    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9e31291-62c6-4f66-af91-c2dc5f2e0a03-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:20:36.729329 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.729281    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Jul 28 02:20:36.735724 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:36.735693    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 02:20:37.132772 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:37.132735    2555 generic.go:356] "Generic (PLEG): container finished" podID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerID="7cc67881cfe13ee22d495b0ea4398afe3f5fe39ceda2e46e5df3d8397bbcf10a" exitCode=2
Jul 28 02:20:37.133226 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:37.132811    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" event={"ID":"a11cf713-5373-4a0a-9813-f1136bb9b315","Type":"ContainerDied","Data":"7cc67881cfe13ee22d495b0ea4398afe3f5fe39ceda2e46e5df3d8397bbcf10a"}
Jul 28 02:20:37.133226 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:37.132848    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"
Jul 28 02:20:37.170589 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:37.170553    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"]
Jul 28 02:20:37.176643 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:37.176616    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a0f617-predictor-69747d6786-zcw7n"]
Jul 28 02:20:37.451151 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:37.451060    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9e31291-62c6-4f66-af91-c2dc5f2e0a03" path="/var/lib/kubelet/pods/f9e31291-62c6-4f66-af91-c2dc5f2e0a03/volumes"
Jul 28 02:20:40.655804 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:40.655780    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:20:40.742182 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:40.742152    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a11cf713-5373-4a0a-9813-f1136bb9b315-kserve-provision-location\") pod \"a11cf713-5373-4a0a-9813-f1136bb9b315\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") "
Jul 28 02:20:40.742342 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:40.742195    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a11cf713-5373-4a0a-9813-f1136bb9b315-cabundle-cert\") pod \"a11cf713-5373-4a0a-9813-f1136bb9b315\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") "
Jul 28 02:20:40.742342 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:40.742252    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a11cf713-5373-4a0a-9813-f1136bb9b315-proxy-tls\") pod \"a11cf713-5373-4a0a-9813-f1136bb9b315\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") "
Jul 28 02:20:40.742342 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:40.742292    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a11cf713-5373-4a0a-9813-f1136bb9b315-kube-rbac-proxy-sar-config\") pod \"a11cf713-5373-4a0a-9813-f1136bb9b315\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") "
Jul 28 02:20:40.742342 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:40.742318    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnwdr\" (UniqueName: \"kubernetes.io/projected/a11cf713-5373-4a0a-9813-f1136bb9b315-kube-api-access-nnwdr\") pod \"a11cf713-5373-4a0a-9813-f1136bb9b315\" (UID: \"a11cf713-5373-4a0a-9813-f1136bb9b315\") "
Jul 28 02:20:40.742603 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:40.742582    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11cf713-5373-4a0a-9813-f1136bb9b315-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a11cf713-5373-4a0a-9813-f1136bb9b315" (UID: "a11cf713-5373-4a0a-9813-f1136bb9b315"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:20:40.742706 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:40.742678    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11cf713-5373-4a0a-9813-f1136bb9b315-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a11cf713-5373-4a0a-9813-f1136bb9b315" (UID: "a11cf713-5373-4a0a-9813-f1136bb9b315"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:20:40.742762 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:40.742723    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11cf713-5373-4a0a-9813-f1136bb9b315-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a11cf713-5373-4a0a-9813-f1136bb9b315" (UID: "a11cf713-5373-4a0a-9813-f1136bb9b315"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:20:40.744462 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:40.744437    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11cf713-5373-4a0a-9813-f1136bb9b315-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a11cf713-5373-4a0a-9813-f1136bb9b315" (UID: "a11cf713-5373-4a0a-9813-f1136bb9b315"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:20:40.744558 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:40.744517    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11cf713-5373-4a0a-9813-f1136bb9b315-kube-api-access-nnwdr" (OuterVolumeSpecName: "kube-api-access-nnwdr") pod "a11cf713-5373-4a0a-9813-f1136bb9b315" (UID: "a11cf713-5373-4a0a-9813-f1136bb9b315"). InnerVolumeSpecName "kube-api-access-nnwdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:20:40.842963 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:40.842932    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a11cf713-5373-4a0a-9813-f1136bb9b315-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:20:40.842963 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:40.842960    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a11cf713-5373-4a0a-9813-f1136bb9b315-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:20:40.842963 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:40.842970    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnwdr\" (UniqueName: \"kubernetes.io/projected/a11cf713-5373-4a0a-9813-f1136bb9b315-kube-api-access-nnwdr\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:20:40.843177 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:40.842981    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a11cf713-5373-4a0a-9813-f1136bb9b315-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:20:40.843177 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:40.842990    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a11cf713-5373-4a0a-9813-f1136bb9b315-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:20:41.147989 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.147902    2555 generic.go:356] "Generic (PLEG): container finished" podID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerID="475debacea1eac000dec7caa8eb4f35f0f876e7313e0fe6490276b855a63c6b0" exitCode=0
Jul 28 02:20:41.147989 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.147960    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"
Jul 28 02:20:41.148171 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.147987    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" event={"ID":"a11cf713-5373-4a0a-9813-f1136bb9b315","Type":"ContainerDied","Data":"475debacea1eac000dec7caa8eb4f35f0f876e7313e0fe6490276b855a63c6b0"}
Jul 28 02:20:41.148171 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.148055    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4" event={"ID":"a11cf713-5373-4a0a-9813-f1136bb9b315","Type":"ContainerDied","Data":"7e2cd88bc8498bb46ce0e9b14d7d0afdf40fa78b765d4a2fb580e6651e5231db"}
Jul 28 02:20:41.148171 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.148094    2555 scope.go:117] "RemoveContainer" containerID="7cc67881cfe13ee22d495b0ea4398afe3f5fe39ceda2e46e5df3d8397bbcf10a"
Jul 28 02:20:41.153663 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.153629    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"]
Jul 28 02:20:41.153861 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.153849    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9e31291-62c6-4f66-af91-c2dc5f2e0a03" containerName="storage-initializer"
Jul 28 02:20:41.153913 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.153862    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9e31291-62c6-4f66-af91-c2dc5f2e0a03" containerName="storage-initializer"
Jul 28 02:20:41.153913 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.153872    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kube-rbac-proxy"
Jul 28 02:20:41.153913 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.153877    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kube-rbac-proxy"
Jul 28 02:20:41.153913 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.153890    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9e31291-62c6-4f66-af91-c2dc5f2e0a03" containerName="storage-initializer"
Jul 28 02:20:41.153913 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.153895    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9e31291-62c6-4f66-af91-c2dc5f2e0a03" containerName="storage-initializer"
Jul 28 02:20:41.153913 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.153906    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="storage-initializer"
Jul 28 02:20:41.153913 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.153912    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="storage-initializer"
Jul 28 02:20:41.154146 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.153918    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kserve-container"
Jul 28 02:20:41.154146 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.153923    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kserve-container"
Jul 28 02:20:41.154146 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.153962    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kserve-container"
Jul 28 02:20:41.154146 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.153968    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9e31291-62c6-4f66-af91-c2dc5f2e0a03" containerName="storage-initializer"
Jul 28 02:20:41.154146 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.153974    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9e31291-62c6-4f66-af91-c2dc5f2e0a03" containerName="storage-initializer"
Jul 28 02:20:41.154146 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.153979    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" containerName="kube-rbac-proxy"
Jul 28 02:20:41.157233 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.157219    2555 scope.go:117] "RemoveContainer" containerID="475debacea1eac000dec7caa8eb4f35f0f876e7313e0fe6490276b855a63c6b0"
Jul 28 02:20:41.158103 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.158075    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:41.160628 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.160605    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-ab280c-predictor-serving-cert\""
Jul 28 02:20:41.160766 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.160739    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:20:41.160951 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.160825    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:20:41.160951 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.160825    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-ab280c-kube-rbac-proxy-sar-config\""
Jul 28 02:20:41.160951 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.160863    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-ab280c-dockercfg-25sbp\""
Jul 28 02:20:41.160951 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.160831    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-ab280c\""
Jul 28 02:20:41.161252 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.161235    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:20:41.166316 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.166294    2555 scope.go:117] "RemoveContainer" containerID="2892014f8416db475fe79ed8e3b80a81ae0a2c30c8c28ce7c1d716a684bf557e"
Jul 28 02:20:41.166427 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.166362    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"]
Jul 28 02:20:41.171069 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.171049    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"]
Jul 28 02:20:41.174071 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.174048    2555 scope.go:117] "RemoveContainer" containerID="7cc67881cfe13ee22d495b0ea4398afe3f5fe39ceda2e46e5df3d8397bbcf10a"
Jul 28 02:20:41.174351 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:20:41.174328    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc67881cfe13ee22d495b0ea4398afe3f5fe39ceda2e46e5df3d8397bbcf10a\": container with ID starting with 7cc67881cfe13ee22d495b0ea4398afe3f5fe39ceda2e46e5df3d8397bbcf10a not found: ID does not exist" containerID="7cc67881cfe13ee22d495b0ea4398afe3f5fe39ceda2e46e5df3d8397bbcf10a"
Jul 28 02:20:41.174405 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.174361    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc67881cfe13ee22d495b0ea4398afe3f5fe39ceda2e46e5df3d8397bbcf10a"} err="failed to get container status \"7cc67881cfe13ee22d495b0ea4398afe3f5fe39ceda2e46e5df3d8397bbcf10a\": rpc error: code = NotFound desc = could not find container \"7cc67881cfe13ee22d495b0ea4398afe3f5fe39ceda2e46e5df3d8397bbcf10a\": container with ID starting with 7cc67881cfe13ee22d495b0ea4398afe3f5fe39ceda2e46e5df3d8397bbcf10a not found: ID does not exist"
Jul 28 02:20:41.174405 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.174380    2555 scope.go:117] "RemoveContainer" containerID="475debacea1eac000dec7caa8eb4f35f0f876e7313e0fe6490276b855a63c6b0"
Jul 28 02:20:41.174752 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:20:41.174734    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475debacea1eac000dec7caa8eb4f35f0f876e7313e0fe6490276b855a63c6b0\": container with ID starting with 475debacea1eac000dec7caa8eb4f35f0f876e7313e0fe6490276b855a63c6b0 not found: ID does not exist" containerID="475debacea1eac000dec7caa8eb4f35f0f876e7313e0fe6490276b855a63c6b0"
Jul 28 02:20:41.174881 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.174758    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475debacea1eac000dec7caa8eb4f35f0f876e7313e0fe6490276b855a63c6b0"} err="failed to get container status \"475debacea1eac000dec7caa8eb4f35f0f876e7313e0fe6490276b855a63c6b0\": rpc error: code = NotFound desc = could not find container \"475debacea1eac000dec7caa8eb4f35f0f876e7313e0fe6490276b855a63c6b0\": container with ID starting with 475debacea1eac000dec7caa8eb4f35f0f876e7313e0fe6490276b855a63c6b0 not found: ID does not exist"
Jul 28 02:20:41.174881 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.174773    2555 scope.go:117] "RemoveContainer" containerID="2892014f8416db475fe79ed8e3b80a81ae0a2c30c8c28ce7c1d716a684bf557e"
Jul 28 02:20:41.175045 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:20:41.175029    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2892014f8416db475fe79ed8e3b80a81ae0a2c30c8c28ce7c1d716a684bf557e\": container with ID starting with 2892014f8416db475fe79ed8e3b80a81ae0a2c30c8c28ce7c1d716a684bf557e not found: ID does not exist" containerID="2892014f8416db475fe79ed8e3b80a81ae0a2c30c8c28ce7c1d716a684bf557e"
Jul 28 02:20:41.175086 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.175050    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2892014f8416db475fe79ed8e3b80a81ae0a2c30c8c28ce7c1d716a684bf557e"} err="failed to get container status \"2892014f8416db475fe79ed8e3b80a81ae0a2c30c8c28ce7c1d716a684bf557e\": rpc error: code = NotFound desc = could not find container \"2892014f8416db475fe79ed8e3b80a81ae0a2c30c8c28ce7c1d716a684bf557e\": container with ID starting with 2892014f8416db475fe79ed8e3b80a81ae0a2c30c8c28ce7c1d716a684bf557e not found: ID does not exist"
Jul 28 02:20:41.177091 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.177067    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a0f617-predictor-5758d4969-69kf4"]
Jul 28 02:20:41.244962 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.244918    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrz59\" (UniqueName: \"kubernetes.io/projected/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kube-api-access-qrz59\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:41.244962 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.244966    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:41.245171 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.245019    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-proxy-tls\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:41.245171 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.245044    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-cabundle-cert\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:41.245171 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.245095    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kserve-provision-location\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:41.345796 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.345756    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-proxy-tls\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:41.345796 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.345803    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-cabundle-cert\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:41.346008 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:20:41.345919    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-serving-cert: secret "isvc-init-fail-ab280c-predictor-serving-cert" not found
Jul 28 02:20:41.346008 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.345923    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kserve-provision-location\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:41.346008 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:20:41.345982    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-proxy-tls podName:8fdcb8ea-d28b-4c4a-bceb-397f08746d5b nodeName:}" failed. No retries permitted until 2026-07-28 02:20:41.845965588 +0000 UTC m=+3288.855483991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-proxy-tls") pod "isvc-init-fail-ab280c-predictor-864cbfc576-rm22t" (UID: "8fdcb8ea-d28b-4c4a-bceb-397f08746d5b") : secret "isvc-init-fail-ab280c-predictor-serving-cert" not found
Jul 28 02:20:41.346156 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.346044    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrz59\" (UniqueName: \"kubernetes.io/projected/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kube-api-access-qrz59\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:41.346156 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.346087    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:41.346302 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.346281    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kserve-provision-location\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:41.346516 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.346500    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-cabundle-cert\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:41.346661 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.346645    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:41.355109 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.355085    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrz59\" (UniqueName: \"kubernetes.io/projected/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kube-api-access-qrz59\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:41.451583 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.451489    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a11cf713-5373-4a0a-9813-f1136bb9b315" path="/var/lib/kubelet/pods/a11cf713-5373-4a0a-9813-f1136bb9b315/volumes"
Jul 28 02:20:41.849016 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:41.848985    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-proxy-tls\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:41.849404 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:20:41.849106    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-serving-cert: secret "isvc-init-fail-ab280c-predictor-serving-cert" not found
Jul 28 02:20:41.849404 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:20:41.849160    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-proxy-tls podName:8fdcb8ea-d28b-4c4a-bceb-397f08746d5b nodeName:}" failed. No retries permitted until 2026-07-28 02:20:42.849143714 +0000 UTC m=+3289.858662118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-proxy-tls") pod "isvc-init-fail-ab280c-predictor-864cbfc576-rm22t" (UID: "8fdcb8ea-d28b-4c4a-bceb-397f08746d5b") : secret "isvc-init-fail-ab280c-predictor-serving-cert" not found
Jul 28 02:20:42.857230 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:42.857191    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-proxy-tls\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:42.859878 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:42.859850    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-proxy-tls\") pod \"isvc-init-fail-ab280c-predictor-864cbfc576-rm22t\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") " pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:42.970517 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:42.970484    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:43.098114 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:43.098085    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"]
Jul 28 02:20:43.100898 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:20:43.100867    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fdcb8ea_d28b_4c4a_bceb_397f08746d5b.slice/crio-0c583d35e0dc5f3843b5b2f91eb55f0fb32bbd4d409d93cf676a948edc336428 WatchSource:0}: Error finding container 0c583d35e0dc5f3843b5b2f91eb55f0fb32bbd4d409d93cf676a948edc336428: Status 404 returned error can't find the container with id 0c583d35e0dc5f3843b5b2f91eb55f0fb32bbd4d409d93cf676a948edc336428
Jul 28 02:20:43.156914 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:43.156887    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t" event={"ID":"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b","Type":"ContainerStarted","Data":"0c583d35e0dc5f3843b5b2f91eb55f0fb32bbd4d409d93cf676a948edc336428"}
Jul 28 02:20:44.161430 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:44.161388    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t" event={"ID":"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b","Type":"ContainerStarted","Data":"131c400093f0a60db6851e41605e4107495162097cb04f864158ea77c8f49780"}
Jul 28 02:20:47.173236 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:47.173210    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ab280c-predictor-864cbfc576-rm22t_8fdcb8ea-d28b-4c4a-bceb-397f08746d5b/storage-initializer/0.log"
Jul 28 02:20:47.173689 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:47.173252    2555 generic.go:356] "Generic (PLEG): container finished" podID="8fdcb8ea-d28b-4c4a-bceb-397f08746d5b" containerID="131c400093f0a60db6851e41605e4107495162097cb04f864158ea77c8f49780" exitCode=1
Jul 28 02:20:47.173689 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:47.173300    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t" event={"ID":"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b","Type":"ContainerDied","Data":"131c400093f0a60db6851e41605e4107495162097cb04f864158ea77c8f49780"}
Jul 28 02:20:48.178490 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:48.178460    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ab280c-predictor-864cbfc576-rm22t_8fdcb8ea-d28b-4c4a-bceb-397f08746d5b/storage-initializer/0.log"
Jul 28 02:20:48.178908 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:48.178561    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t" event={"ID":"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b","Type":"ContainerStarted","Data":"ba0baa3a5df793b56ea572c4e56b7d6a45f9edbdc4d017de29340fee2c617014"}
Jul 28 02:20:51.163405 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.163370    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"]
Jul 28 02:20:51.163851 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.163658    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t" podUID="8fdcb8ea-d28b-4c4a-bceb-397f08746d5b" containerName="storage-initializer" containerID="cri-o://ba0baa3a5df793b56ea572c4e56b7d6a45f9edbdc4d017de29340fee2c617014" gracePeriod=30
Jul 28 02:20:51.190732 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.190696    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ab280c-predictor-864cbfc576-rm22t_8fdcb8ea-d28b-4c4a-bceb-397f08746d5b/storage-initializer/1.log"
Jul 28 02:20:51.191166 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.191147    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ab280c-predictor-864cbfc576-rm22t_8fdcb8ea-d28b-4c4a-bceb-397f08746d5b/storage-initializer/0.log"
Jul 28 02:20:51.191282 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.191184    2555 generic.go:356] "Generic (PLEG): container finished" podID="8fdcb8ea-d28b-4c4a-bceb-397f08746d5b" containerID="ba0baa3a5df793b56ea572c4e56b7d6a45f9edbdc4d017de29340fee2c617014" exitCode=1
Jul 28 02:20:51.191282 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.191257    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t" event={"ID":"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b","Type":"ContainerDied","Data":"ba0baa3a5df793b56ea572c4e56b7d6a45f9edbdc4d017de29340fee2c617014"}
Jul 28 02:20:51.191392 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.191325    2555 scope.go:117] "RemoveContainer" containerID="131c400093f0a60db6851e41605e4107495162097cb04f864158ea77c8f49780"
Jul 28 02:20:51.299347 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.299324    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ab280c-predictor-864cbfc576-rm22t_8fdcb8ea-d28b-4c4a-bceb-397f08746d5b/storage-initializer/1.log"
Jul 28 02:20:51.302584 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.302564    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:51.320035 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.320002    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kserve-provision-location\") pod \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") "
Jul 28 02:20:51.320181 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.320063    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kube-rbac-proxy-sar-config\") pod \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") "
Jul 28 02:20:51.320181 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.320088    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-proxy-tls\") pod \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") "
Jul 28 02:20:51.320181 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.320117    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrz59\" (UniqueName: \"kubernetes.io/projected/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kube-api-access-qrz59\") pod \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") "
Jul 28 02:20:51.320181 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.320133    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-cabundle-cert\") pod \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\" (UID: \"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b\") "
Jul 28 02:20:51.320381 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.320271    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8fdcb8ea-d28b-4c4a-bceb-397f08746d5b" (UID: "8fdcb8ea-d28b-4c4a-bceb-397f08746d5b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:20:51.320487 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.320467    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8fdcb8ea-d28b-4c4a-bceb-397f08746d5b" (UID: "8fdcb8ea-d28b-4c4a-bceb-397f08746d5b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:20:51.320632 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.320606    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8fdcb8ea-d28b-4c4a-bceb-397f08746d5b" (UID: "8fdcb8ea-d28b-4c4a-bceb-397f08746d5b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:20:51.322829 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.322801    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8fdcb8ea-d28b-4c4a-bceb-397f08746d5b" (UID: "8fdcb8ea-d28b-4c4a-bceb-397f08746d5b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:20:51.322924 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.322895    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kube-api-access-qrz59" (OuterVolumeSpecName: "kube-api-access-qrz59") pod "8fdcb8ea-d28b-4c4a-bceb-397f08746d5b" (UID: "8fdcb8ea-d28b-4c4a-bceb-397f08746d5b"). InnerVolumeSpecName "kube-api-access-qrz59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:20:51.421485 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.421390    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:20:51.421485 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.421427    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrz59\" (UniqueName: \"kubernetes.io/projected/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kube-api-access-qrz59\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:20:51.421485 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.421438    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:20:51.421485 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.421446    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:20:51.421485 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:51.421455    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:20:52.196100 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:52.196071    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ab280c-predictor-864cbfc576-rm22t_8fdcb8ea-d28b-4c4a-bceb-397f08746d5b/storage-initializer/1.log"
Jul 28 02:20:52.196479 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:52.196143    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t" event={"ID":"8fdcb8ea-d28b-4c4a-bceb-397f08746d5b","Type":"ContainerDied","Data":"0c583d35e0dc5f3843b5b2f91eb55f0fb32bbd4d409d93cf676a948edc336428"}
Jul 28 02:20:52.196479 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:52.196171    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"
Jul 28 02:20:52.196479 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:52.196191    2555 scope.go:117] "RemoveContainer" containerID="ba0baa3a5df793b56ea572c4e56b7d6a45f9edbdc4d017de29340fee2c617014"
Jul 28 02:20:52.228621 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:52.228589    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"]
Jul 28 02:20:52.232575 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:52.232550    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ab280c-predictor-864cbfc576-rm22t"]
Jul 28 02:20:53.451366 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:53.451337    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fdcb8ea-d28b-4c4a-bceb-397f08746d5b" path="/var/lib/kubelet/pods/8fdcb8ea-d28b-4c4a-bceb-397f08746d5b/volumes"
Jul 28 02:20:56.373020 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.372984    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"]
Jul 28 02:20:56.373382 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.373243    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fdcb8ea-d28b-4c4a-bceb-397f08746d5b" containerName="storage-initializer"
Jul 28 02:20:56.373382 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.373254    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fdcb8ea-d28b-4c4a-bceb-397f08746d5b" containerName="storage-initializer"
Jul 28 02:20:56.373382 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.373270    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fdcb8ea-d28b-4c4a-bceb-397f08746d5b" containerName="storage-initializer"
Jul 28 02:20:56.373382 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.373276    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fdcb8ea-d28b-4c4a-bceb-397f08746d5b" containerName="storage-initializer"
Jul 28 02:20:56.373382 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.373318    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fdcb8ea-d28b-4c4a-bceb-397f08746d5b" containerName="storage-initializer"
Jul 28 02:20:56.373563 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.373393    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fdcb8ea-d28b-4c4a-bceb-397f08746d5b" containerName="storage-initializer"
Jul 28 02:20:56.377258 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.377237    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.379977 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.379946    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:20:56.380317 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.380265    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jul 28 02:20:56.380317 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.380290    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jul 28 02:20:56.380317 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.380313    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:20:56.380599 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.380312    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:20:56.380599 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.380269    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:20:56.381384 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.381368    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:20:56.397154 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.397126    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"]
Jul 28 02:20:56.459211 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.459176    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e33999-cb79-4a1f-9504-531d27d4bfd2-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-75pvf\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.459211 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.459208    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e33999-cb79-4a1f-9504-531d27d4bfd2-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-75pvf\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.459467 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.459237    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-984wr\" (UniqueName: \"kubernetes.io/projected/09e33999-cb79-4a1f-9504-531d27d4bfd2-kube-api-access-984wr\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-75pvf\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.459467 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.459266    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e33999-cb79-4a1f-9504-531d27d4bfd2-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-75pvf\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.459467 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.459349    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e33999-cb79-4a1f-9504-531d27d4bfd2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-75pvf\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.560579 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.560517    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e33999-cb79-4a1f-9504-531d27d4bfd2-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-75pvf\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.560579 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.560579    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e33999-cb79-4a1f-9504-531d27d4bfd2-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-75pvf\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.560828 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.560606    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-984wr\" (UniqueName: \"kubernetes.io/projected/09e33999-cb79-4a1f-9504-531d27d4bfd2-kube-api-access-984wr\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-75pvf\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.560828 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.560630    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e33999-cb79-4a1f-9504-531d27d4bfd2-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-75pvf\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.560828 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.560671    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e33999-cb79-4a1f-9504-531d27d4bfd2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-75pvf\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.561120 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.561094    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e33999-cb79-4a1f-9504-531d27d4bfd2-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-75pvf\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.561308 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.561286    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e33999-cb79-4a1f-9504-531d27d4bfd2-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-75pvf\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.561373 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.561326    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e33999-cb79-4a1f-9504-531d27d4bfd2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-75pvf\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.563211 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.563181    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e33999-cb79-4a1f-9504-531d27d4bfd2-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-75pvf\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.569116 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.569090    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-984wr\" (UniqueName: \"kubernetes.io/projected/09e33999-cb79-4a1f-9504-531d27d4bfd2-kube-api-access-984wr\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-75pvf\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.687095 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.687013    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:20:56.820956 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:56.820922    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"]
Jul 28 02:20:56.824645 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:20:56.824617    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e33999_cb79_4a1f_9504_531d27d4bfd2.slice/crio-f762bb69d17fb36e4291fb6f4042a07f6dd3fd7fbf616004c7eedfc0aaeeb700 WatchSource:0}: Error finding container f762bb69d17fb36e4291fb6f4042a07f6dd3fd7fbf616004c7eedfc0aaeeb700: Status 404 returned error can't find the container with id f762bb69d17fb36e4291fb6f4042a07f6dd3fd7fbf616004c7eedfc0aaeeb700
Jul 28 02:20:57.215041 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:57.215006    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" event={"ID":"09e33999-cb79-4a1f-9504-531d27d4bfd2","Type":"ContainerStarted","Data":"f617fe3ffc0c551677a9aa521cd4a9f8b6cb01cf806188b4ca9d711e38454c30"}
Jul 28 02:20:57.215254 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:20:57.215084    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" event={"ID":"09e33999-cb79-4a1f-9504-531d27d4bfd2","Type":"ContainerStarted","Data":"f762bb69d17fb36e4291fb6f4042a07f6dd3fd7fbf616004c7eedfc0aaeeb700"}
Jul 28 02:21:01.230345 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:21:01.230309    2555 generic.go:356] "Generic (PLEG): container finished" podID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerID="f617fe3ffc0c551677a9aa521cd4a9f8b6cb01cf806188b4ca9d711e38454c30" exitCode=0
Jul 28 02:21:01.230778 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:21:01.230380    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" event={"ID":"09e33999-cb79-4a1f-9504-531d27d4bfd2","Type":"ContainerDied","Data":"f617fe3ffc0c551677a9aa521cd4a9f8b6cb01cf806188b4ca9d711e38454c30"}
Jul 28 02:21:01.231589 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:21:01.231572    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 02:21:24.331131 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:21:24.331093    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" event={"ID":"09e33999-cb79-4a1f-9504-531d27d4bfd2","Type":"ContainerStarted","Data":"f3813ca4da80e053a9193a1baf26ea8f5d833f495db205cf8b2088d273a57183"}
Jul 28 02:21:24.331669 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:21:24.331170    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" event={"ID":"09e33999-cb79-4a1f-9504-531d27d4bfd2","Type":"ContainerStarted","Data":"11f70d9d9551ebe544583b6d34483fd1f72e3358391b1b8e06d71c9f5db66d5f"}
Jul 28 02:21:24.331669 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:21:24.331404    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:21:24.352912 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:21:24.352851    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" podStartSLOduration=2.0976593550000002 podStartE2EDuration="28.35281056s" totalImagesPullingTime="22.481052541s" totalInitContainerRuntime="3.774098664s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:20:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:21:24.352780736 +0000 UTC m=+3331.362299162" watchObservedRunningTime="2026-07-28 02:21:24.35281056 +0000 UTC m=+3331.362328985"
Jul 28 02:21:25.334595 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:21:25.334558    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:21:25.335687 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:21:25.335658    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 02:21:26.344583 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:21:26.344505    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 02:21:31.348690 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:21:31.348662    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:21:31.349211 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:21:31.349184    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 02:21:41.350005 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:21:41.349966    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 02:21:51.349654 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:21:51.349614    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 02:22:01.349796 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:01.349758    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 02:22:11.349258 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:11.349218    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 02:22:21.349990 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:21.349945    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 02:22:31.349452 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:31.349413    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 02:22:40.447707 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:40.447672    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:22:46.526003 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:46.525919    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"]
Jul 28 02:22:46.526512 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:46.526224    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kserve-container" containerID="cri-o://11f70d9d9551ebe544583b6d34483fd1f72e3358391b1b8e06d71c9f5db66d5f" gracePeriod=30
Jul 28 02:22:46.526512 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:46.526255    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kube-rbac-proxy" containerID="cri-o://f3813ca4da80e053a9193a1baf26ea8f5d833f495db205cf8b2088d273a57183" gracePeriod=30
Jul 28 02:22:47.629331 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:47.629300    2555 generic.go:356] "Generic (PLEG): container finished" podID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerID="f3813ca4da80e053a9193a1baf26ea8f5d833f495db205cf8b2088d273a57183" exitCode=2
Jul 28 02:22:47.629735 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:47.629348    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" event={"ID":"09e33999-cb79-4a1f-9504-531d27d4bfd2","Type":"ContainerDied","Data":"f3813ca4da80e053a9193a1baf26ea8f5d833f495db205cf8b2088d273a57183"}
Jul 28 02:22:50.448208 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:50.448175    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 02:22:51.375779 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.375754    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:22:51.407676 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.407649    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e33999-cb79-4a1f-9504-531d27d4bfd2-kube-rbac-proxy-sar-config\") pod \"09e33999-cb79-4a1f-9504-531d27d4bfd2\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") "
Jul 28 02:22:51.407862 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.407687    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-984wr\" (UniqueName: \"kubernetes.io/projected/09e33999-cb79-4a1f-9504-531d27d4bfd2-kube-api-access-984wr\") pod \"09e33999-cb79-4a1f-9504-531d27d4bfd2\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") "
Jul 28 02:22:51.407862 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.407713    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e33999-cb79-4a1f-9504-531d27d4bfd2-kserve-provision-location\") pod \"09e33999-cb79-4a1f-9504-531d27d4bfd2\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") "
Jul 28 02:22:51.407993 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.407909    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e33999-cb79-4a1f-9504-531d27d4bfd2-cabundle-cert\") pod \"09e33999-cb79-4a1f-9504-531d27d4bfd2\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") "
Jul 28 02:22:51.407993 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.407988    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e33999-cb79-4a1f-9504-531d27d4bfd2-proxy-tls\") pod \"09e33999-cb79-4a1f-9504-531d27d4bfd2\" (UID: \"09e33999-cb79-4a1f-9504-531d27d4bfd2\") "
Jul 28 02:22:51.408110 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.408027    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e33999-cb79-4a1f-9504-531d27d4bfd2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09e33999-cb79-4a1f-9504-531d27d4bfd2" (UID: "09e33999-cb79-4a1f-9504-531d27d4bfd2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:22:51.408110 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.408048    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e33999-cb79-4a1f-9504-531d27d4bfd2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09e33999-cb79-4a1f-9504-531d27d4bfd2" (UID: "09e33999-cb79-4a1f-9504-531d27d4bfd2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:22:51.408209 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.408188    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e33999-cb79-4a1f-9504-531d27d4bfd2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:22:51.408262 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.408207    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e33999-cb79-4a1f-9504-531d27d4bfd2-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:22:51.408318 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.408268    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e33999-cb79-4a1f-9504-531d27d4bfd2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09e33999-cb79-4a1f-9504-531d27d4bfd2" (UID: "09e33999-cb79-4a1f-9504-531d27d4bfd2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:22:51.410067 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.410011    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e33999-cb79-4a1f-9504-531d27d4bfd2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09e33999-cb79-4a1f-9504-531d27d4bfd2" (UID: "09e33999-cb79-4a1f-9504-531d27d4bfd2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:22:51.410067 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.410039    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e33999-cb79-4a1f-9504-531d27d4bfd2-kube-api-access-984wr" (OuterVolumeSpecName: "kube-api-access-984wr") pod "09e33999-cb79-4a1f-9504-531d27d4bfd2" (UID: "09e33999-cb79-4a1f-9504-531d27d4bfd2"). InnerVolumeSpecName "kube-api-access-984wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:22:51.508898 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.508854    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e33999-cb79-4a1f-9504-531d27d4bfd2-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:22:51.508898 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.508891    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e33999-cb79-4a1f-9504-531d27d4bfd2-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:22:51.508898 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.508900    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-984wr\" (UniqueName: \"kubernetes.io/projected/09e33999-cb79-4a1f-9504-531d27d4bfd2-kube-api-access-984wr\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:22:51.644916 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.644874    2555 generic.go:356] "Generic (PLEG): container finished" podID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerID="11f70d9d9551ebe544583b6d34483fd1f72e3358391b1b8e06d71c9f5db66d5f" exitCode=0
Jul 28 02:22:51.645078 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.644937    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" event={"ID":"09e33999-cb79-4a1f-9504-531d27d4bfd2","Type":"ContainerDied","Data":"11f70d9d9551ebe544583b6d34483fd1f72e3358391b1b8e06d71c9f5db66d5f"}
Jul 28 02:22:51.645078 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.644951    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"
Jul 28 02:22:51.645078 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.645033    2555 scope.go:117] "RemoveContainer" containerID="f3813ca4da80e053a9193a1baf26ea8f5d833f495db205cf8b2088d273a57183"
Jul 28 02:22:51.645234 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.645022    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" event={"ID":"09e33999-cb79-4a1f-9504-531d27d4bfd2","Type":"ContainerDied","Data":"f762bb69d17fb36e4291fb6f4042a07f6dd3fd7fbf616004c7eedfc0aaeeb700"}
Jul 28 02:22:51.653260 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.653241    2555 scope.go:117] "RemoveContainer" containerID="11f70d9d9551ebe544583b6d34483fd1f72e3358391b1b8e06d71c9f5db66d5f"
Jul 28 02:22:51.659286 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.659258    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"]
Jul 28 02:22:51.659504 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.659492    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kserve-container"
Jul 28 02:22:51.659572 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.659506    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kserve-container"
Jul 28 02:22:51.659572 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.659550    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="storage-initializer"
Jul 28 02:22:51.659572 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.659558    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="storage-initializer"
Jul 28 02:22:51.659572 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.659570    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kube-rbac-proxy"
Jul 28 02:22:51.659716 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.659578    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kube-rbac-proxy"
Jul 28 02:22:51.659716 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.659623    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kserve-container"
Jul 28 02:22:51.659716 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.659629    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kube-rbac-proxy"
Jul 28 02:22:51.660816 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.660793    2555 scope.go:117] "RemoveContainer" containerID="f617fe3ffc0c551677a9aa521cd4a9f8b6cb01cf806188b4ca9d711e38454c30"
Jul 28 02:22:51.662236 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.662219    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:51.665795 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.665641    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jul 28 02:22:51.665795 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.665697    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jul 28 02:22:51.665795 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.665757    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:22:51.665994 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.665763    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:22:51.666400 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.666385    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:22:51.666703 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.666688    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:22:51.666773 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.666741    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:22:51.670076 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.670032    2555 scope.go:117] "RemoveContainer" containerID="f3813ca4da80e053a9193a1baf26ea8f5d833f495db205cf8b2088d273a57183"
Jul 28 02:22:51.670414 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:22:51.670381    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3813ca4da80e053a9193a1baf26ea8f5d833f495db205cf8b2088d273a57183\": container with ID starting with f3813ca4da80e053a9193a1baf26ea8f5d833f495db205cf8b2088d273a57183 not found: ID does not exist" containerID="f3813ca4da80e053a9193a1baf26ea8f5d833f495db205cf8b2088d273a57183"
Jul 28 02:22:51.670486 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.670418    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3813ca4da80e053a9193a1baf26ea8f5d833f495db205cf8b2088d273a57183"} err="failed to get container status \"f3813ca4da80e053a9193a1baf26ea8f5d833f495db205cf8b2088d273a57183\": rpc error: code = NotFound desc = could not find container \"f3813ca4da80e053a9193a1baf26ea8f5d833f495db205cf8b2088d273a57183\": container with ID starting with f3813ca4da80e053a9193a1baf26ea8f5d833f495db205cf8b2088d273a57183 not found: ID does not exist"
Jul 28 02:22:51.670486 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.670442    2555 scope.go:117] "RemoveContainer" containerID="11f70d9d9551ebe544583b6d34483fd1f72e3358391b1b8e06d71c9f5db66d5f"
Jul 28 02:22:51.670786 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:22:51.670766    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f70d9d9551ebe544583b6d34483fd1f72e3358391b1b8e06d71c9f5db66d5f\": container with ID starting with 11f70d9d9551ebe544583b6d34483fd1f72e3358391b1b8e06d71c9f5db66d5f not found: ID does not exist" containerID="11f70d9d9551ebe544583b6d34483fd1f72e3358391b1b8e06d71c9f5db66d5f"
Jul 28 02:22:51.670865 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.670795    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f70d9d9551ebe544583b6d34483fd1f72e3358391b1b8e06d71c9f5db66d5f"} err="failed to get container status \"11f70d9d9551ebe544583b6d34483fd1f72e3358391b1b8e06d71c9f5db66d5f\": rpc error: code = NotFound desc = could not find container \"11f70d9d9551ebe544583b6d34483fd1f72e3358391b1b8e06d71c9f5db66d5f\": container with ID starting with 11f70d9d9551ebe544583b6d34483fd1f72e3358391b1b8e06d71c9f5db66d5f not found: ID does not exist"
Jul 28 02:22:51.670865 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.670815    2555 scope.go:117] "RemoveContainer" containerID="f617fe3ffc0c551677a9aa521cd4a9f8b6cb01cf806188b4ca9d711e38454c30"
Jul 28 02:22:51.671104 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:22:51.671085    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f617fe3ffc0c551677a9aa521cd4a9f8b6cb01cf806188b4ca9d711e38454c30\": container with ID starting with f617fe3ffc0c551677a9aa521cd4a9f8b6cb01cf806188b4ca9d711e38454c30 not found: ID does not exist" containerID="f617fe3ffc0c551677a9aa521cd4a9f8b6cb01cf806188b4ca9d711e38454c30"
Jul 28 02:22:51.671155 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.671109    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f617fe3ffc0c551677a9aa521cd4a9f8b6cb01cf806188b4ca9d711e38454c30"} err="failed to get container status \"f617fe3ffc0c551677a9aa521cd4a9f8b6cb01cf806188b4ca9d711e38454c30\": rpc error: code = NotFound desc = could not find container \"f617fe3ffc0c551677a9aa521cd4a9f8b6cb01cf806188b4ca9d711e38454c30\": container with ID starting with f617fe3ffc0c551677a9aa521cd4a9f8b6cb01cf806188b4ca9d711e38454c30 not found: ID does not exist"
Jul 28 02:22:51.671758 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.671735    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"]
Jul 28 02:22:51.673586 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.673566    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"]
Jul 28 02:22:51.677802 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.677781    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf"]
Jul 28 02:22:51.710472 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.710433    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a08212c-40f1-46ba-84d9-5891499df55d-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-g79fz\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:51.710685 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.710480    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a08212c-40f1-46ba-84d9-5891499df55d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-g79fz\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:51.710685 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.710514    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a08212c-40f1-46ba-84d9-5891499df55d-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-g79fz\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:51.710685 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.710580    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48w82\" (UniqueName: \"kubernetes.io/projected/3a08212c-40f1-46ba-84d9-5891499df55d-kube-api-access-48w82\") pod \"isvc-predictive-xgboost-predictor-695f978978-g79fz\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:51.710685 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.710614    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a08212c-40f1-46ba-84d9-5891499df55d-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-g79fz\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:51.811129 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.811093    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48w82\" (UniqueName: \"kubernetes.io/projected/3a08212c-40f1-46ba-84d9-5891499df55d-kube-api-access-48w82\") pod \"isvc-predictive-xgboost-predictor-695f978978-g79fz\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:51.811332 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.811133    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a08212c-40f1-46ba-84d9-5891499df55d-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-g79fz\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:51.811332 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.811179    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a08212c-40f1-46ba-84d9-5891499df55d-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-g79fz\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:51.811332 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.811202    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a08212c-40f1-46ba-84d9-5891499df55d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-g79fz\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:51.811332 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.811231    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a08212c-40f1-46ba-84d9-5891499df55d-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-g79fz\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:51.811789 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.811743    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a08212c-40f1-46ba-84d9-5891499df55d-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-g79fz\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:51.812122 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.812090    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a08212c-40f1-46ba-84d9-5891499df55d-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-g79fz\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:51.812240 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.812156    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a08212c-40f1-46ba-84d9-5891499df55d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-g79fz\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:51.814042 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.814022    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a08212c-40f1-46ba-84d9-5891499df55d-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-g79fz\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:51.819501 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.819469    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48w82\" (UniqueName: \"kubernetes.io/projected/3a08212c-40f1-46ba-84d9-5891499df55d-kube-api-access-48w82\") pod \"isvc-predictive-xgboost-predictor-695f978978-g79fz\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:51.974744 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:51.974646    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:52.103339 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:52.103314    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"]
Jul 28 02:22:52.105777 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:22:52.105753    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a08212c_40f1_46ba_84d9_5891499df55d.slice/crio-8b639976128e6b9ed46d34cd6109a7da5142a550056fa0020d54cbed56a06a68 WatchSource:0}: Error finding container 8b639976128e6b9ed46d34cd6109a7da5142a550056fa0020d54cbed56a06a68: Status 404 returned error can't find the container with id 8b639976128e6b9ed46d34cd6109a7da5142a550056fa0020d54cbed56a06a68
Jul 28 02:22:52.345965 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:52.345916    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-75pvf" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": context deadline exceeded"
Jul 28 02:22:52.650542 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:52.650424    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" event={"ID":"3a08212c-40f1-46ba-84d9-5891499df55d","Type":"ContainerStarted","Data":"98c20bc498cea251a96c09476812255b16e9956115c608d958e9dc6678582638"}
Jul 28 02:22:52.650542 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:52.650495    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" event={"ID":"3a08212c-40f1-46ba-84d9-5891499df55d","Type":"ContainerStarted","Data":"8b639976128e6b9ed46d34cd6109a7da5142a550056fa0020d54cbed56a06a68"}
Jul 28 02:22:53.451557 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:53.451503    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09e33999-cb79-4a1f-9504-531d27d4bfd2" path="/var/lib/kubelet/pods/09e33999-cb79-4a1f-9504-531d27d4bfd2/volumes"
Jul 28 02:22:56.670689 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:56.670650    2555 generic.go:356] "Generic (PLEG): container finished" podID="3a08212c-40f1-46ba-84d9-5891499df55d" containerID="98c20bc498cea251a96c09476812255b16e9956115c608d958e9dc6678582638" exitCode=0
Jul 28 02:22:56.671081 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:56.670702    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" event={"ID":"3a08212c-40f1-46ba-84d9-5891499df55d","Type":"ContainerDied","Data":"98c20bc498cea251a96c09476812255b16e9956115c608d958e9dc6678582638"}
Jul 28 02:22:57.676742 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:57.676711    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" event={"ID":"3a08212c-40f1-46ba-84d9-5891499df55d","Type":"ContainerStarted","Data":"e733d663f6ccc4848d10a932b3c98e1c4b5bf44ddf7e82bfc7b09e443d6dd561"}
Jul 28 02:22:57.677134 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:57.676778    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" event={"ID":"3a08212c-40f1-46ba-84d9-5891499df55d","Type":"ContainerStarted","Data":"20204d169c31924abeb20979dd90ee6f3f460bcce6eb0a7e5b29bdc202c1bde1"}
Jul 28 02:22:57.677134 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:57.676981    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:57.696584 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:57.696517    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" podStartSLOduration=2.81900065 podStartE2EDuration="6.696500709s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.877500059s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:22:57.695342788 +0000 UTC m=+3424.704861237" watchObservedRunningTime="2026-07-28 02:22:57.696500709 +0000 UTC m=+3424.706019135"
Jul 28 02:22:58.680308 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:58.680277    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:22:58.681507 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:58.681476    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 02:22:59.684327 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:22:59.684288    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 02:23:04.688578 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:23:04.688541    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:23:04.689158 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:23:04.689129    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 02:23:14.688996 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:23:14.688955    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 02:23:24.689685 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:23:24.689645    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 02:23:34.689441 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:23:34.689403    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 02:23:44.689042 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:23:44.688998    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 02:23:54.689061 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:23:54.689020    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 02:24:04.689090 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:04.689050    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 02:24:13.451542 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:13.451493    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:24:21.785715 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:21.785623    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"]
Jul 28 02:24:21.786139 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:21.785919    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kube-rbac-proxy" containerID="cri-o://e733d663f6ccc4848d10a932b3c98e1c4b5bf44ddf7e82bfc7b09e443d6dd561" gracePeriod=30
Jul 28 02:24:21.786139 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:21.785908    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kserve-container" containerID="cri-o://20204d169c31924abeb20979dd90ee6f3f460bcce6eb0a7e5b29bdc202c1bde1" gracePeriod=30
Jul 28 02:24:21.975587 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:21.975554    2555 generic.go:356] "Generic (PLEG): container finished" podID="3a08212c-40f1-46ba-84d9-5891499df55d" containerID="e733d663f6ccc4848d10a932b3c98e1c4b5bf44ddf7e82bfc7b09e443d6dd561" exitCode=2
Jul 28 02:24:21.975753 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:21.975604    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" event={"ID":"3a08212c-40f1-46ba-84d9-5891499df55d","Type":"ContainerDied","Data":"e733d663f6ccc4848d10a932b3c98e1c4b5bf44ddf7e82bfc7b09e443d6dd561"}
Jul 28 02:24:23.449602 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:23.449553    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 02:24:24.684969 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:24.684914    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Jul 28 02:24:26.219852 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.219830    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:24:26.287540 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.287490    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a08212c-40f1-46ba-84d9-5891499df55d-kserve-provision-location\") pod \"3a08212c-40f1-46ba-84d9-5891499df55d\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") "
Jul 28 02:24:26.287727 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.287551    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a08212c-40f1-46ba-84d9-5891499df55d-proxy-tls\") pod \"3a08212c-40f1-46ba-84d9-5891499df55d\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") "
Jul 28 02:24:26.287727 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.287579    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a08212c-40f1-46ba-84d9-5891499df55d-kube-rbac-proxy-sar-config\") pod \"3a08212c-40f1-46ba-84d9-5891499df55d\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") "
Jul 28 02:24:26.287727 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.287614    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a08212c-40f1-46ba-84d9-5891499df55d-cabundle-cert\") pod \"3a08212c-40f1-46ba-84d9-5891499df55d\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") "
Jul 28 02:24:26.287727 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.287658    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48w82\" (UniqueName: \"kubernetes.io/projected/3a08212c-40f1-46ba-84d9-5891499df55d-kube-api-access-48w82\") pod \"3a08212c-40f1-46ba-84d9-5891499df55d\" (UID: \"3a08212c-40f1-46ba-84d9-5891499df55d\") "
Jul 28 02:24:26.287952 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.287865    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a08212c-40f1-46ba-84d9-5891499df55d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a08212c-40f1-46ba-84d9-5891499df55d" (UID: "3a08212c-40f1-46ba-84d9-5891499df55d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:24:26.288029 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.288001    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a08212c-40f1-46ba-84d9-5891499df55d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a08212c-40f1-46ba-84d9-5891499df55d" (UID: "3a08212c-40f1-46ba-84d9-5891499df55d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:24:26.288078 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.288014    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a08212c-40f1-46ba-84d9-5891499df55d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a08212c-40f1-46ba-84d9-5891499df55d" (UID: "3a08212c-40f1-46ba-84d9-5891499df55d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:24:26.289827 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.289804    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a08212c-40f1-46ba-84d9-5891499df55d-kube-api-access-48w82" (OuterVolumeSpecName: "kube-api-access-48w82") pod "3a08212c-40f1-46ba-84d9-5891499df55d" (UID: "3a08212c-40f1-46ba-84d9-5891499df55d"). InnerVolumeSpecName "kube-api-access-48w82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:24:26.289917 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.289899    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a08212c-40f1-46ba-84d9-5891499df55d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a08212c-40f1-46ba-84d9-5891499df55d" (UID: "3a08212c-40f1-46ba-84d9-5891499df55d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:24:26.388720 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.388638    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a08212c-40f1-46ba-84d9-5891499df55d-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:24:26.388720 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.388676    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a08212c-40f1-46ba-84d9-5891499df55d-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:24:26.388720 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.388685    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a08212c-40f1-46ba-84d9-5891499df55d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:24:26.388720 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.388693    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a08212c-40f1-46ba-84d9-5891499df55d-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:24:26.388720 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.388704    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48w82\" (UniqueName: \"kubernetes.io/projected/3a08212c-40f1-46ba-84d9-5891499df55d-kube-api-access-48w82\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:24:26.939496 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.939458    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"]
Jul 28 02:24:26.939723 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.939711    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kserve-container"
Jul 28 02:24:26.939784 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.939725    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kserve-container"
Jul 28 02:24:26.939784 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.939745    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kube-rbac-proxy"
Jul 28 02:24:26.939784 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.939752    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kube-rbac-proxy"
Jul 28 02:24:26.939784 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.939760    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="storage-initializer"
Jul 28 02:24:26.939784 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.939766    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="storage-initializer"
Jul 28 02:24:26.939962 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.939804    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kube-rbac-proxy"
Jul 28 02:24:26.939962 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.939813    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" containerName="kserve-container"
Jul 28 02:24:26.941813 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.941798    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:26.944662 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.944643    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 28 02:24:26.944793 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.944673    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 28 02:24:26.951925 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.951902    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"]
Jul 28 02:24:26.994603 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.994559    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e938d13-edae-4262-a712-d006cfa72003-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:26.994603 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.994610    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e938d13-edae-4262-a712-d006cfa72003-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:26.994747 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.994660    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e938d13-edae-4262-a712-d006cfa72003-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:26.994747 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.994702    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e938d13-edae-4262-a712-d006cfa72003-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:26.994814 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:26.994754    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvbp5\" (UniqueName: \"kubernetes.io/projected/4e938d13-edae-4262-a712-d006cfa72003-kube-api-access-dvbp5\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:27.003040 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.003016    2555 generic.go:356] "Generic (PLEG): container finished" podID="3a08212c-40f1-46ba-84d9-5891499df55d" containerID="20204d169c31924abeb20979dd90ee6f3f460bcce6eb0a7e5b29bdc202c1bde1" exitCode=0
Jul 28 02:24:27.003138 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.003075    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"
Jul 28 02:24:27.003138 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.003087    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" event={"ID":"3a08212c-40f1-46ba-84d9-5891499df55d","Type":"ContainerDied","Data":"20204d169c31924abeb20979dd90ee6f3f460bcce6eb0a7e5b29bdc202c1bde1"}
Jul 28 02:24:27.003138 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.003129    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz" event={"ID":"3a08212c-40f1-46ba-84d9-5891499df55d","Type":"ContainerDied","Data":"8b639976128e6b9ed46d34cd6109a7da5142a550056fa0020d54cbed56a06a68"}
Jul 28 02:24:27.003241 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.003154    2555 scope.go:117] "RemoveContainer" containerID="e733d663f6ccc4848d10a932b3c98e1c4b5bf44ddf7e82bfc7b09e443d6dd561"
Jul 28 02:24:27.011287 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.011268    2555 scope.go:117] "RemoveContainer" containerID="20204d169c31924abeb20979dd90ee6f3f460bcce6eb0a7e5b29bdc202c1bde1"
Jul 28 02:24:27.018291 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.018277    2555 scope.go:117] "RemoveContainer" containerID="98c20bc498cea251a96c09476812255b16e9956115c608d958e9dc6678582638"
Jul 28 02:24:27.023996 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.023977    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"]
Jul 28 02:24:27.025356 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.025341    2555 scope.go:117] "RemoveContainer" containerID="e733d663f6ccc4848d10a932b3c98e1c4b5bf44ddf7e82bfc7b09e443d6dd561"
Jul 28 02:24:27.025667 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:24:27.025623    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e733d663f6ccc4848d10a932b3c98e1c4b5bf44ddf7e82bfc7b09e443d6dd561\": container with ID starting with e733d663f6ccc4848d10a932b3c98e1c4b5bf44ddf7e82bfc7b09e443d6dd561 not found: ID does not exist" containerID="e733d663f6ccc4848d10a932b3c98e1c4b5bf44ddf7e82bfc7b09e443d6dd561"
Jul 28 02:24:27.025823 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.025668    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e733d663f6ccc4848d10a932b3c98e1c4b5bf44ddf7e82bfc7b09e443d6dd561"} err="failed to get container status \"e733d663f6ccc4848d10a932b3c98e1c4b5bf44ddf7e82bfc7b09e443d6dd561\": rpc error: code = NotFound desc = could not find container \"e733d663f6ccc4848d10a932b3c98e1c4b5bf44ddf7e82bfc7b09e443d6dd561\": container with ID starting with e733d663f6ccc4848d10a932b3c98e1c4b5bf44ddf7e82bfc7b09e443d6dd561 not found: ID does not exist"
Jul 28 02:24:27.025823 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.025718    2555 scope.go:117] "RemoveContainer" containerID="20204d169c31924abeb20979dd90ee6f3f460bcce6eb0a7e5b29bdc202c1bde1"
Jul 28 02:24:27.026251 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:24:27.026229    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20204d169c31924abeb20979dd90ee6f3f460bcce6eb0a7e5b29bdc202c1bde1\": container with ID starting with 20204d169c31924abeb20979dd90ee6f3f460bcce6eb0a7e5b29bdc202c1bde1 not found: ID does not exist" containerID="20204d169c31924abeb20979dd90ee6f3f460bcce6eb0a7e5b29bdc202c1bde1"
Jul 28 02:24:27.026320 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.026258    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20204d169c31924abeb20979dd90ee6f3f460bcce6eb0a7e5b29bdc202c1bde1"} err="failed to get container status \"20204d169c31924abeb20979dd90ee6f3f460bcce6eb0a7e5b29bdc202c1bde1\": rpc error: code = NotFound desc = could not find container \"20204d169c31924abeb20979dd90ee6f3f460bcce6eb0a7e5b29bdc202c1bde1\": container with ID starting with 20204d169c31924abeb20979dd90ee6f3f460bcce6eb0a7e5b29bdc202c1bde1 not found: ID does not exist"
Jul 28 02:24:27.026320 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.026275    2555 scope.go:117] "RemoveContainer" containerID="98c20bc498cea251a96c09476812255b16e9956115c608d958e9dc6678582638"
Jul 28 02:24:27.026591 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:24:27.026566    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c20bc498cea251a96c09476812255b16e9956115c608d958e9dc6678582638\": container with ID starting with 98c20bc498cea251a96c09476812255b16e9956115c608d958e9dc6678582638 not found: ID does not exist" containerID="98c20bc498cea251a96c09476812255b16e9956115c608d958e9dc6678582638"
Jul 28 02:24:27.026687 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.026597    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c20bc498cea251a96c09476812255b16e9956115c608d958e9dc6678582638"} err="failed to get container status \"98c20bc498cea251a96c09476812255b16e9956115c608d958e9dc6678582638\": rpc error: code = NotFound desc = could not find container \"98c20bc498cea251a96c09476812255b16e9956115c608d958e9dc6678582638\": container with ID starting with 98c20bc498cea251a96c09476812255b16e9956115c608d958e9dc6678582638 not found: ID does not exist"
Jul 28 02:24:27.027279 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.027261    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-g79fz"]
Jul 28 02:24:27.095762 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.095741    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvbp5\" (UniqueName: \"kubernetes.io/projected/4e938d13-edae-4262-a712-d006cfa72003-kube-api-access-dvbp5\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:27.095887 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.095774    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e938d13-edae-4262-a712-d006cfa72003-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:27.095887 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.095795    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e938d13-edae-4262-a712-d006cfa72003-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:27.095887 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.095821    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e938d13-edae-4262-a712-d006cfa72003-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:27.095887 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.095853    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e938d13-edae-4262-a712-d006cfa72003-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:27.096196 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.096175    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e938d13-edae-4262-a712-d006cfa72003-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:27.096676 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.096656    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e938d13-edae-4262-a712-d006cfa72003-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:27.096733 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.096656    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e938d13-edae-4262-a712-d006cfa72003-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:27.098227 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.098208    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e938d13-edae-4262-a712-d006cfa72003-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:27.103457 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.103427    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvbp5\" (UniqueName: \"kubernetes.io/projected/4e938d13-edae-4262-a712-d006cfa72003-kube-api-access-dvbp5\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:27.252647 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.252604    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:27.380285 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.380259    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"]
Jul 28 02:24:27.382787 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:24:27.382751    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e938d13_edae_4262_a712_d006cfa72003.slice/crio-2a137e0f6e21cdf3eb16a29c4c99f63614902b75c4a7a18bf5708b45525b2520 WatchSource:0}: Error finding container 2a137e0f6e21cdf3eb16a29c4c99f63614902b75c4a7a18bf5708b45525b2520: Status 404 returned error can't find the container with id 2a137e0f6e21cdf3eb16a29c4c99f63614902b75c4a7a18bf5708b45525b2520
Jul 28 02:24:27.451789 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:27.451762    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a08212c-40f1-46ba-84d9-5891499df55d" path="/var/lib/kubelet/pods/3a08212c-40f1-46ba-84d9-5891499df55d/volumes"
Jul 28 02:24:28.008784 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:28.008752    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" event={"ID":"4e938d13-edae-4262-a712-d006cfa72003","Type":"ContainerStarted","Data":"166e17bae557f59cde3db1707ddf929d90ef6b80637f0d674895e439ff04bf40"}
Jul 28 02:24:28.008954 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:28.008808    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" event={"ID":"4e938d13-edae-4262-a712-d006cfa72003","Type":"ContainerStarted","Data":"2a137e0f6e21cdf3eb16a29c4c99f63614902b75c4a7a18bf5708b45525b2520"}
Jul 28 02:24:32.023711 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:32.023673    2555 generic.go:356] "Generic (PLEG): container finished" podID="4e938d13-edae-4262-a712-d006cfa72003" containerID="166e17bae557f59cde3db1707ddf929d90ef6b80637f0d674895e439ff04bf40" exitCode=0
Jul 28 02:24:32.024089 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:32.023750    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" event={"ID":"4e938d13-edae-4262-a712-d006cfa72003","Type":"ContainerDied","Data":"166e17bae557f59cde3db1707ddf929d90ef6b80637f0d674895e439ff04bf40"}
Jul 28 02:24:33.029192 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:33.029155    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" event={"ID":"4e938d13-edae-4262-a712-d006cfa72003","Type":"ContainerStarted","Data":"d99562799cade25bcade47066e4396b96f4f857424c72dd90faa31c7aec32dac"}
Jul 28 02:24:33.029620 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:33.029220    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" event={"ID":"4e938d13-edae-4262-a712-d006cfa72003","Type":"ContainerStarted","Data":"73c5392905a90074fc48285d76e26e2f698fb59d0a6886dac7f77539a4442735"}
Jul 28 02:24:33.029620 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:33.029548    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:33.029703 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:33.029678    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:33.030993 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:33.030968    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 02:24:33.049473 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:33.049430    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" podStartSLOduration=3.395863574 podStartE2EDuration="7.049414712s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.653551138s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:24:33.049076077 +0000 UTC m=+3520.058594503" watchObservedRunningTime="2026-07-28 02:24:33.049414712 +0000 UTC m=+3520.058933118"
Jul 28 02:24:34.032463 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:34.032421    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 02:24:35.036148 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:35.036108    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 02:24:40.040727 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:40.040694    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:24:40.041363 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:40.041333    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 02:24:50.042159 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:24:50.042116    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 02:25:00.041884 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:25:00.041841    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 02:25:10.041738 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:25:10.041695    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 02:25:20.041376 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:25:20.041332    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 02:25:30.042025 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:25:30.041984    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 02:25:40.041477 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:25:40.041432    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 02:25:50.041748 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:25:50.041675    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:25:57.014216 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:25:57.014176    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"]
Jul 28 02:25:57.014681 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:25:57.014442    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kserve-container" containerID="cri-o://73c5392905a90074fc48285d76e26e2f698fb59d0a6886dac7f77539a4442735" gracePeriod=30
Jul 28 02:25:57.014681 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:25:57.014467    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kube-rbac-proxy" containerID="cri-o://d99562799cade25bcade47066e4396b96f4f857424c72dd90faa31c7aec32dac" gracePeriod=30
Jul 28 02:25:57.318681 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:25:57.318600    2555 generic.go:356] "Generic (PLEG): container finished" podID="4e938d13-edae-4262-a712-d006cfa72003" containerID="d99562799cade25bcade47066e4396b96f4f857424c72dd90faa31c7aec32dac" exitCode=2
Jul 28 02:25:57.318821 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:25:57.318673    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" event={"ID":"4e938d13-edae-4262-a712-d006cfa72003","Type":"ContainerDied","Data":"d99562799cade25bcade47066e4396b96f4f857424c72dd90faa31c7aec32dac"}
Jul 28 02:26:00.036751 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:00.036709    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Jul 28 02:26:00.042072 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:00.042039    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 02:26:02.177070 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.177031    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"]
Jul 28 02:26:02.180302 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.180284    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.182913 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.182888    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 28 02:26:02.182913 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.182906    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 28 02:26:02.191976 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.191950    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"]
Jul 28 02:26:02.287014 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.286948    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.287014 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.286991    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp7nx\" (UniqueName: \"kubernetes.io/projected/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kube-api-access-fp7nx\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.287014 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.287012    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.287229 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.287037    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.287229 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.287095    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.338614 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.338582    2555 generic.go:356] "Generic (PLEG): container finished" podID="4e938d13-edae-4262-a712-d006cfa72003" containerID="73c5392905a90074fc48285d76e26e2f698fb59d0a6886dac7f77539a4442735" exitCode=0
Jul 28 02:26:02.338774 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.338658    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" event={"ID":"4e938d13-edae-4262-a712-d006cfa72003","Type":"ContainerDied","Data":"73c5392905a90074fc48285d76e26e2f698fb59d0a6886dac7f77539a4442735"}
Jul 28 02:26:02.388044 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.388013    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.388139 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.388055    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.388139 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.388115    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.388261 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.388143    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp7nx\" (UniqueName: \"kubernetes.io/projected/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kube-api-access-fp7nx\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.388261 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.388167    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.388646 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.388610    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.388830 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.388814    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.388882 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.388827    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.390733 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.390711    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.397835 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.397815    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp7nx\" (UniqueName: \"kubernetes.io/projected/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kube-api-access-fp7nx\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.490038 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.489987    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:02.662755 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.662732    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:26:02.690325 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.690286    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvbp5\" (UniqueName: \"kubernetes.io/projected/4e938d13-edae-4262-a712-d006cfa72003-kube-api-access-dvbp5\") pod \"4e938d13-edae-4262-a712-d006cfa72003\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") "
Jul 28 02:26:02.690484 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.690357    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e938d13-edae-4262-a712-d006cfa72003-cabundle-cert\") pod \"4e938d13-edae-4262-a712-d006cfa72003\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") "
Jul 28 02:26:02.690484 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.690394    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e938d13-edae-4262-a712-d006cfa72003-kube-rbac-proxy-sar-config\") pod \"4e938d13-edae-4262-a712-d006cfa72003\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") "
Jul 28 02:26:02.690484 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.690429    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e938d13-edae-4262-a712-d006cfa72003-kserve-provision-location\") pod \"4e938d13-edae-4262-a712-d006cfa72003\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") "
Jul 28 02:26:02.690484 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.690462    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e938d13-edae-4262-a712-d006cfa72003-proxy-tls\") pod \"4e938d13-edae-4262-a712-d006cfa72003\" (UID: \"4e938d13-edae-4262-a712-d006cfa72003\") "
Jul 28 02:26:02.690839 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.690805    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e938d13-edae-4262-a712-d006cfa72003-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e938d13-edae-4262-a712-d006cfa72003" (UID: "4e938d13-edae-4262-a712-d006cfa72003"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:26:02.690904 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.690829    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e938d13-edae-4262-a712-d006cfa72003-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e938d13-edae-4262-a712-d006cfa72003" (UID: "4e938d13-edae-4262-a712-d006cfa72003"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:26:02.690904 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.690812    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e938d13-edae-4262-a712-d006cfa72003-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4e938d13-edae-4262-a712-d006cfa72003" (UID: "4e938d13-edae-4262-a712-d006cfa72003"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:26:02.692698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.692676    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e938d13-edae-4262-a712-d006cfa72003-kube-api-access-dvbp5" (OuterVolumeSpecName: "kube-api-access-dvbp5") pod "4e938d13-edae-4262-a712-d006cfa72003" (UID: "4e938d13-edae-4262-a712-d006cfa72003"). InnerVolumeSpecName "kube-api-access-dvbp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:26:02.692777 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.692695    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e938d13-edae-4262-a712-d006cfa72003-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e938d13-edae-4262-a712-d006cfa72003" (UID: "4e938d13-edae-4262-a712-d006cfa72003"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:26:02.791699 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.791630    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e938d13-edae-4262-a712-d006cfa72003-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:26:02.791699 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.791652    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e938d13-edae-4262-a712-d006cfa72003-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:26:02.791699 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.791661    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e938d13-edae-4262-a712-d006cfa72003-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:26:02.791699 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.791671    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e938d13-edae-4262-a712-d006cfa72003-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:26:02.791699 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.791681    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvbp5\" (UniqueName: \"kubernetes.io/projected/4e938d13-edae-4262-a712-d006cfa72003-kube-api-access-dvbp5\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:26:02.832160 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:02.832127    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"]
Jul 28 02:26:02.835135 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:26:02.835108    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5042bd3_b4f5_4f4d_a415_b7ac9b9861db.slice/crio-384b6ecae467af0897cfb7e68fa85852aec1f5a2628875a84d3b48db368d425d WatchSource:0}: Error finding container 384b6ecae467af0897cfb7e68fa85852aec1f5a2628875a84d3b48db368d425d: Status 404 returned error can't find the container with id 384b6ecae467af0897cfb7e68fa85852aec1f5a2628875a84d3b48db368d425d
Jul 28 02:26:03.344101 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:03.344063    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" event={"ID":"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db","Type":"ContainerStarted","Data":"79a05c66bffef63b4b865bb71d0aed8316b6d10c9b44ae74d9ef3dab275a3151"}
Jul 28 02:26:03.344637 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:03.344133    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" event={"ID":"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db","Type":"ContainerStarted","Data":"384b6ecae467af0897cfb7e68fa85852aec1f5a2628875a84d3b48db368d425d"}
Jul 28 02:26:03.345963 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:03.345938    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4" event={"ID":"4e938d13-edae-4262-a712-d006cfa72003","Type":"ContainerDied","Data":"2a137e0f6e21cdf3eb16a29c4c99f63614902b75c4a7a18bf5708b45525b2520"}
Jul 28 02:26:03.346102 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:03.345996    2555 scope.go:117] "RemoveContainer" containerID="d99562799cade25bcade47066e4396b96f4f857424c72dd90faa31c7aec32dac"
Jul 28 02:26:03.346102 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:03.345947    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"
Jul 28 02:26:03.354668 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:03.354643    2555 scope.go:117] "RemoveContainer" containerID="73c5392905a90074fc48285d76e26e2f698fb59d0a6886dac7f77539a4442735"
Jul 28 02:26:03.363396 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:03.363368    2555 scope.go:117] "RemoveContainer" containerID="166e17bae557f59cde3db1707ddf929d90ef6b80637f0d674895e439ff04bf40"
Jul 28 02:26:03.382403 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:03.382374    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"]
Jul 28 02:26:03.386034 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:03.386009    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-mstg4"]
Jul 28 02:26:03.451890 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:03.451859    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e938d13-edae-4262-a712-d006cfa72003" path="/var/lib/kubelet/pods/4e938d13-edae-4262-a712-d006cfa72003/volumes"
Jul 28 02:26:07.361234 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:07.361200    2555 generic.go:356] "Generic (PLEG): container finished" podID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerID="79a05c66bffef63b4b865bb71d0aed8316b6d10c9b44ae74d9ef3dab275a3151" exitCode=0
Jul 28 02:26:07.361641 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:07.361242    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" event={"ID":"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db","Type":"ContainerDied","Data":"79a05c66bffef63b4b865bb71d0aed8316b6d10c9b44ae74d9ef3dab275a3151"}
Jul 28 02:26:08.367874 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:08.367833    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" event={"ID":"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db","Type":"ContainerStarted","Data":"a0276a6d39515be424d43ed2eef5847df2ccc85dd4fb626b7f87e72a715d66e6"}
Jul 28 02:26:08.368252 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:08.367896    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" event={"ID":"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db","Type":"ContainerStarted","Data":"8f3e72aacc00874bee74479764cf0589c641571cb57979239b20c9c334138313"}
Jul 28 02:26:08.368252 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:08.368109    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:08.368342 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:08.368247    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:08.388949 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:08.388899    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" podStartSLOduration=2.583589713 podStartE2EDuration="6.38888173s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.805292017s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:26:08.388079804 +0000 UTC m=+3615.397598229" watchObservedRunningTime="2026-07-28 02:26:08.38888173 +0000 UTC m=+3615.398400158"
Jul 28 02:26:14.377292 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:14.377263    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:26:44.378123 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:44.378075    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 02:26:54.377875 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:54.377832    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 02:26:55.166251 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:26:55.166220    2555 scope.go:117] "RemoveContainer" containerID="a524784876b8b24c92b404df6559a5bf0feeb768b1f40ba58adc221efbca8837"
Jul 28 02:27:04.378592 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:04.378542    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 02:27:14.378746 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:14.378652    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 02:27:24.463948 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:24.463913    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:27:33.169100 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:33.169069    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"]
Jul 28 02:27:33.169506 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:33.169293    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerName="kserve-container" containerID="cri-o://8f3e72aacc00874bee74479764cf0589c641571cb57979239b20c9c334138313" gracePeriod=30
Jul 28 02:27:33.169506 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:33.169371    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerName="kube-rbac-proxy" containerID="cri-o://a0276a6d39515be424d43ed2eef5847df2ccc85dd4fb626b7f87e72a715d66e6" gracePeriod=30
Jul 28 02:27:33.653092 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:33.653059    2555 generic.go:356] "Generic (PLEG): container finished" podID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerID="a0276a6d39515be424d43ed2eef5847df2ccc85dd4fb626b7f87e72a715d66e6" exitCode=2
Jul 28 02:27:33.653254 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:33.653111    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" event={"ID":"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db","Type":"ContainerDied","Data":"a0276a6d39515be424d43ed2eef5847df2ccc85dd4fb626b7f87e72a715d66e6"}
Jul 28 02:27:34.372367 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:34.372328    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Jul 28 02:27:34.378196 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:34.378170    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 02:27:37.911402 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:37.911373    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:27:37.976899 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:37.976867    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kserve-provision-location\") pod \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") "
Jul 28 02:27:37.976899 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:37.976913    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-cabundle-cert\") pod \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") "
Jul 28 02:27:37.977167 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:37.976944    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kube-rbac-proxy-sar-config\") pod \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") "
Jul 28 02:27:37.977167 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:37.976964    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-proxy-tls\") pod \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") "
Jul 28 02:27:37.977167 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:37.976982    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp7nx\" (UniqueName: \"kubernetes.io/projected/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kube-api-access-fp7nx\") pod \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\" (UID: \"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db\") "
Jul 28 02:27:37.977327 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:37.977279    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" (UID: "d5042bd3-b4f5-4f4d-a415-b7ac9b9861db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:27:37.977327 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:37.977310    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" (UID: "d5042bd3-b4f5-4f4d-a415-b7ac9b9861db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:27:37.977327 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:37.977285    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" (UID: "d5042bd3-b4f5-4f4d-a415-b7ac9b9861db"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:27:37.979318 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:37.979290    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kube-api-access-fp7nx" (OuterVolumeSpecName: "kube-api-access-fp7nx") pod "d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" (UID: "d5042bd3-b4f5-4f4d-a415-b7ac9b9861db"). InnerVolumeSpecName "kube-api-access-fp7nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:27:37.979318 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:37.979305    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" (UID: "d5042bd3-b4f5-4f4d-a415-b7ac9b9861db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:27:38.077468 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.077371    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:27:38.077468 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.077403    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:27:38.077468 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.077416    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:27:38.077468 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.077429    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp7nx\" (UniqueName: \"kubernetes.io/projected/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kube-api-access-fp7nx\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:27:38.077468 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.077440    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:27:38.339932 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.339853    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"]
Jul 28 02:27:38.340113 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.340101    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kserve-container"
Jul 28 02:27:38.340160 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.340115    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kserve-container"
Jul 28 02:27:38.340160 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.340122    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kube-rbac-proxy"
Jul 28 02:27:38.340160 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.340128    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kube-rbac-proxy"
Jul 28 02:27:38.340160 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.340135    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerName="kube-rbac-proxy"
Jul 28 02:27:38.340160 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.340140    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerName="kube-rbac-proxy"
Jul 28 02:27:38.340160 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.340153    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerName="kserve-container"
Jul 28 02:27:38.340160 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.340158    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerName="kserve-container"
Jul 28 02:27:38.340379 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.340167    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="storage-initializer"
Jul 28 02:27:38.340379 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.340172    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="storage-initializer"
Jul 28 02:27:38.340379 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.340180    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerName="storage-initializer"
Jul 28 02:27:38.340379 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.340186    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerName="storage-initializer"
Jul 28 02:27:38.340379 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.340225    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kube-rbac-proxy"
Jul 28 02:27:38.340379 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.340231    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerName="kserve-container"
Jul 28 02:27:38.340379 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.340238    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerName="kube-rbac-proxy"
Jul 28 02:27:38.340379 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.340244    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e938d13-edae-4262-a712-d006cfa72003" containerName="kserve-container"
Jul 28 02:27:38.343429 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.343412    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:38.345987 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.345962    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 28 02:27:38.346088 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.345963    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 28 02:27:38.355806 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.355783    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"]
Jul 28 02:27:38.379687 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.379652    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f98v\" (UniqueName: \"kubernetes.io/projected/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kube-api-access-7f98v\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:38.379687 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.379689    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:38.379961 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.379714    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4ffc608-6e63-40a4-8049-e3721b05fcaa-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:38.379961 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.379740    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:38.379961 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.379761    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4ffc608-6e63-40a4-8049-e3721b05fcaa-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:38.480192 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.480154    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f98v\" (UniqueName: \"kubernetes.io/projected/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kube-api-access-7f98v\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:38.480192 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.480193    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:38.480408 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.480225    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4ffc608-6e63-40a4-8049-e3721b05fcaa-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:38.480408 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.480251    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:38.480408 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.480277    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4ffc608-6e63-40a4-8049-e3721b05fcaa-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:38.480408 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:27:38.480397    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Jul 28 02:27:38.480637 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:27:38.480474    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ffc608-6e63-40a4-8049-e3721b05fcaa-proxy-tls podName:b4ffc608-6e63-40a4-8049-e3721b05fcaa nodeName:}" failed. No retries permitted until 2026-07-28 02:27:38.98045316 +0000 UTC m=+3705.989971567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b4ffc608-6e63-40a4-8049-e3721b05fcaa-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" (UID: "b4ffc608-6e63-40a4-8049-e3721b05fcaa") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Jul 28 02:27:38.480637 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.480627    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:38.480936 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.480919    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4ffc608-6e63-40a4-8049-e3721b05fcaa-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:38.480987 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.480971    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:38.491648 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.491621    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f98v\" (UniqueName: \"kubernetes.io/projected/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kube-api-access-7f98v\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:38.673838 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.673746    2555 generic.go:356] "Generic (PLEG): container finished" podID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" containerID="8f3e72aacc00874bee74479764cf0589c641571cb57979239b20c9c334138313" exitCode=0
Jul 28 02:27:38.673838 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.673792    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" event={"ID":"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db","Type":"ContainerDied","Data":"8f3e72aacc00874bee74479764cf0589c641571cb57979239b20c9c334138313"}
Jul 28 02:27:38.673838 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.673809    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"
Jul 28 02:27:38.674131 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.673855    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw" event={"ID":"d5042bd3-b4f5-4f4d-a415-b7ac9b9861db","Type":"ContainerDied","Data":"384b6ecae467af0897cfb7e68fa85852aec1f5a2628875a84d3b48db368d425d"}
Jul 28 02:27:38.674131 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.673891    2555 scope.go:117] "RemoveContainer" containerID="a0276a6d39515be424d43ed2eef5847df2ccc85dd4fb626b7f87e72a715d66e6"
Jul 28 02:27:38.682305 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.682288    2555 scope.go:117] "RemoveContainer" containerID="8f3e72aacc00874bee74479764cf0589c641571cb57979239b20c9c334138313"
Jul 28 02:27:38.689360 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.689338    2555 scope.go:117] "RemoveContainer" containerID="79a05c66bffef63b4b865bb71d0aed8316b6d10c9b44ae74d9ef3dab275a3151"
Jul 28 02:27:38.696848 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.696827    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"]
Jul 28 02:27:38.696919 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.696900    2555 scope.go:117] "RemoveContainer" containerID="a0276a6d39515be424d43ed2eef5847df2ccc85dd4fb626b7f87e72a715d66e6"
Jul 28 02:27:38.697177 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:27:38.697160    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0276a6d39515be424d43ed2eef5847df2ccc85dd4fb626b7f87e72a715d66e6\": container with ID starting with a0276a6d39515be424d43ed2eef5847df2ccc85dd4fb626b7f87e72a715d66e6 not found: ID does not exist" containerID="a0276a6d39515be424d43ed2eef5847df2ccc85dd4fb626b7f87e72a715d66e6"
Jul 28 02:27:38.697219 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.697186    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0276a6d39515be424d43ed2eef5847df2ccc85dd4fb626b7f87e72a715d66e6"} err="failed to get container status \"a0276a6d39515be424d43ed2eef5847df2ccc85dd4fb626b7f87e72a715d66e6\": rpc error: code = NotFound desc = could not find container \"a0276a6d39515be424d43ed2eef5847df2ccc85dd4fb626b7f87e72a715d66e6\": container with ID starting with a0276a6d39515be424d43ed2eef5847df2ccc85dd4fb626b7f87e72a715d66e6 not found: ID does not exist"
Jul 28 02:27:38.697219 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.697205    2555 scope.go:117] "RemoveContainer" containerID="8f3e72aacc00874bee74479764cf0589c641571cb57979239b20c9c334138313"
Jul 28 02:27:38.697446 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:27:38.697430    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3e72aacc00874bee74479764cf0589c641571cb57979239b20c9c334138313\": container with ID starting with 8f3e72aacc00874bee74479764cf0589c641571cb57979239b20c9c334138313 not found: ID does not exist" containerID="8f3e72aacc00874bee74479764cf0589c641571cb57979239b20c9c334138313"
Jul 28 02:27:38.697483 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.697454    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3e72aacc00874bee74479764cf0589c641571cb57979239b20c9c334138313"} err="failed to get container status \"8f3e72aacc00874bee74479764cf0589c641571cb57979239b20c9c334138313\": rpc error: code = NotFound desc = could not find container \"8f3e72aacc00874bee74479764cf0589c641571cb57979239b20c9c334138313\": container with ID starting with 8f3e72aacc00874bee74479764cf0589c641571cb57979239b20c9c334138313 not found: ID does not exist"
Jul 28 02:27:38.697483 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.697470    2555 scope.go:117] "RemoveContainer" containerID="79a05c66bffef63b4b865bb71d0aed8316b6d10c9b44ae74d9ef3dab275a3151"
Jul 28 02:27:38.697708 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:27:38.697690    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a05c66bffef63b4b865bb71d0aed8316b6d10c9b44ae74d9ef3dab275a3151\": container with ID starting with 79a05c66bffef63b4b865bb71d0aed8316b6d10c9b44ae74d9ef3dab275a3151 not found: ID does not exist" containerID="79a05c66bffef63b4b865bb71d0aed8316b6d10c9b44ae74d9ef3dab275a3151"
Jul 28 02:27:38.697761 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.697715    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a05c66bffef63b4b865bb71d0aed8316b6d10c9b44ae74d9ef3dab275a3151"} err="failed to get container status \"79a05c66bffef63b4b865bb71d0aed8316b6d10c9b44ae74d9ef3dab275a3151\": rpc error: code = NotFound desc = could not find container \"79a05c66bffef63b4b865bb71d0aed8316b6d10c9b44ae74d9ef3dab275a3151\": container with ID starting with 79a05c66bffef63b4b865bb71d0aed8316b6d10c9b44ae74d9ef3dab275a3151 not found: ID does not exist"
Jul 28 02:27:38.700356 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.700335    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-nqzqw"]
Jul 28 02:27:38.984132 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.984030    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4ffc608-6e63-40a4-8049-e3721b05fcaa-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:38.986554 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:38.986511    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4ffc608-6e63-40a4-8049-e3721b05fcaa-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:39.252855 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:39.252816    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:39.384001 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:39.383970    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"]
Jul 28 02:27:39.387106 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:27:39.387078    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ffc608_6e63_40a4_8049_e3721b05fcaa.slice/crio-7d20b9a5c939bb201573d82c683a6d8ede82e9268953e677c8bd924078fbc353 WatchSource:0}: Error finding container 7d20b9a5c939bb201573d82c683a6d8ede82e9268953e677c8bd924078fbc353: Status 404 returned error can't find the container with id 7d20b9a5c939bb201573d82c683a6d8ede82e9268953e677c8bd924078fbc353
Jul 28 02:27:39.452825 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:39.452798    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5042bd3-b4f5-4f4d-a415-b7ac9b9861db" path="/var/lib/kubelet/pods/d5042bd3-b4f5-4f4d-a415-b7ac9b9861db/volumes"
Jul 28 02:27:39.679558 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:39.679440    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" event={"ID":"b4ffc608-6e63-40a4-8049-e3721b05fcaa","Type":"ContainerStarted","Data":"48eed156c197e3ee0d4e769a61b8bd0928c17ac64e004114223c1179e0ebf3f5"}
Jul 28 02:27:39.679558 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:39.679499    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" event={"ID":"b4ffc608-6e63-40a4-8049-e3721b05fcaa","Type":"ContainerStarted","Data":"7d20b9a5c939bb201573d82c683a6d8ede82e9268953e677c8bd924078fbc353"}
Jul 28 02:27:43.694073 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:43.694037    2555 generic.go:356] "Generic (PLEG): container finished" podID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerID="48eed156c197e3ee0d4e769a61b8bd0928c17ac64e004114223c1179e0ebf3f5" exitCode=0
Jul 28 02:27:43.694430 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:43.694080    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" event={"ID":"b4ffc608-6e63-40a4-8049-e3721b05fcaa","Type":"ContainerDied","Data":"48eed156c197e3ee0d4e769a61b8bd0928c17ac64e004114223c1179e0ebf3f5"}
Jul 28 02:27:44.699535 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:44.699484    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" event={"ID":"b4ffc608-6e63-40a4-8049-e3721b05fcaa","Type":"ContainerStarted","Data":"4748f67eef7339f1e49bd55e685167f84a4701ad5090f351e527d00fe8b6c630"}
Jul 28 02:27:44.699923 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:44.699574    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" event={"ID":"b4ffc608-6e63-40a4-8049-e3721b05fcaa","Type":"ContainerStarted","Data":"e27a0f90286052e4e94bda8cd92ed545b9a8c84a101dff5cdff79da29421401c"}
Jul 28 02:27:44.699923 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:44.699842    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:44.699923 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:44.699904    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:27:44.722857 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:44.722806    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" podStartSLOduration=3.062411108 podStartE2EDuration="6.722791476s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.660380368s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:27:44.721914658 +0000 UTC m=+3711.731433083" watchObservedRunningTime="2026-07-28 02:27:44.722791476 +0000 UTC m=+3711.732309901"
Jul 28 02:27:50.709037 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:27:50.709009    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:28:20.709924 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:28:20.709885    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.53:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 02:28:30.710178 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:28:30.710141    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.53:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 02:28:40.710455 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:28:40.710420    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.53:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 02:28:50.710101 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:28:50.710022    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.53:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 02:29:00.762625 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:00.762598    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:29:08.434040 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:08.434006    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"]
Jul 28 02:29:08.434497 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:08.434246    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerName="kserve-container" containerID="cri-o://e27a0f90286052e4e94bda8cd92ed545b9a8c84a101dff5cdff79da29421401c" gracePeriod=30
Jul 28 02:29:08.434497 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:08.434326    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerName="kube-rbac-proxy" containerID="cri-o://4748f67eef7339f1e49bd55e685167f84a4701ad5090f351e527d00fe8b6c630" gracePeriod=30
Jul 28 02:29:08.988106 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:08.988073    2555 generic.go:356] "Generic (PLEG): container finished" podID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerID="4748f67eef7339f1e49bd55e685167f84a4701ad5090f351e527d00fe8b6c630" exitCode=2
Jul 28 02:29:08.988282 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:08.988126    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" event={"ID":"b4ffc608-6e63-40a4-8049-e3721b05fcaa","Type":"ContainerDied","Data":"4748f67eef7339f1e49bd55e685167f84a4701ad5090f351e527d00fe8b6c630"}
Jul 28 02:29:10.704450 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:10.704399    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Jul 28 02:29:10.710059 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:10.710024    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.53:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 02:29:13.004042 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.004011    2555 generic.go:356] "Generic (PLEG): container finished" podID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerID="e27a0f90286052e4e94bda8cd92ed545b9a8c84a101dff5cdff79da29421401c" exitCode=0
Jul 28 02:29:13.004484 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.004087    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" event={"ID":"b4ffc608-6e63-40a4-8049-e3721b05fcaa","Type":"ContainerDied","Data":"e27a0f90286052e4e94bda8cd92ed545b9a8c84a101dff5cdff79da29421401c"}
Jul 28 02:29:13.074560 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.074536    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:29:13.153416 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.153338    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kube-rbac-proxy-sar-config\") pod \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") "
Jul 28 02:29:13.153416 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.153377    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kserve-provision-location\") pod \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") "
Jul 28 02:29:13.153416 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.153409    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4ffc608-6e63-40a4-8049-e3721b05fcaa-proxy-tls\") pod \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") "
Jul 28 02:29:13.153666 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.153560    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4ffc608-6e63-40a4-8049-e3721b05fcaa-cabundle-cert\") pod \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") "
Jul 28 02:29:13.153666 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.153621    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f98v\" (UniqueName: \"kubernetes.io/projected/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kube-api-access-7f98v\") pod \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\" (UID: \"b4ffc608-6e63-40a4-8049-e3721b05fcaa\") "
Jul 28 02:29:13.153754 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.153741    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b4ffc608-6e63-40a4-8049-e3721b05fcaa" (UID: "b4ffc608-6e63-40a4-8049-e3721b05fcaa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:29:13.153839 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.153741    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b4ffc608-6e63-40a4-8049-e3721b05fcaa" (UID: "b4ffc608-6e63-40a4-8049-e3721b05fcaa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:29:13.153839 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.153805    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:29:13.153957 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.153917    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ffc608-6e63-40a4-8049-e3721b05fcaa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b4ffc608-6e63-40a4-8049-e3721b05fcaa" (UID: "b4ffc608-6e63-40a4-8049-e3721b05fcaa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:29:13.155673 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.155650    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4ffc608-6e63-40a4-8049-e3721b05fcaa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b4ffc608-6e63-40a4-8049-e3721b05fcaa" (UID: "b4ffc608-6e63-40a4-8049-e3721b05fcaa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:29:13.155849 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.155821    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kube-api-access-7f98v" (OuterVolumeSpecName: "kube-api-access-7f98v") pod "b4ffc608-6e63-40a4-8049-e3721b05fcaa" (UID: "b4ffc608-6e63-40a4-8049-e3721b05fcaa"). InnerVolumeSpecName "kube-api-access-7f98v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:29:13.254388 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.254347    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4ffc608-6e63-40a4-8049-e3721b05fcaa-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:29:13.254388 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.254381    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4ffc608-6e63-40a4-8049-e3721b05fcaa-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:29:13.254388 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.254394    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f98v\" (UniqueName: \"kubernetes.io/projected/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kube-api-access-7f98v\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:29:13.254675 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.254406    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4ffc608-6e63-40a4-8049-e3721b05fcaa-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:29:13.603999 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.603958    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"]
Jul 28 02:29:13.604293 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.604275    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerName="kube-rbac-proxy"
Jul 28 02:29:13.604381 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.604296    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerName="kube-rbac-proxy"
Jul 28 02:29:13.604381 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.604313    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerName="storage-initializer"
Jul 28 02:29:13.604381 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.604321    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerName="storage-initializer"
Jul 28 02:29:13.604381 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.604336    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerName="kserve-container"
Jul 28 02:29:13.604381 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.604344    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerName="kserve-container"
Jul 28 02:29:13.604692 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.604416    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerName="kserve-container"
Jul 28 02:29:13.604692 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.604427    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" containerName="kube-rbac-proxy"
Jul 28 02:29:13.607700 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.607679    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:13.610585 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.610560    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jul 28 02:29:13.610722 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.610703    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jul 28 02:29:13.619764 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.619743    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"]
Jul 28 02:29:13.657258 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.657234    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/033b364b-845f-4e4c-b38b-f8e47d81f15e-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:13.657376 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.657274    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/033b364b-845f-4e4c-b38b-f8e47d81f15e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:13.657376 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.657307    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/033b364b-845f-4e4c-b38b-f8e47d81f15e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:13.657376 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.657323    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/033b364b-845f-4e4c-b38b-f8e47d81f15e-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:13.657376 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.657354    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jww2\" (UniqueName: \"kubernetes.io/projected/033b364b-845f-4e4c-b38b-f8e47d81f15e-kube-api-access-4jww2\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:13.757671 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.757639    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/033b364b-845f-4e4c-b38b-f8e47d81f15e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:13.757839 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.757680    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/033b364b-845f-4e4c-b38b-f8e47d81f15e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:13.757839 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.757697    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/033b364b-845f-4e4c-b38b-f8e47d81f15e-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:13.757839 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.757733    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jww2\" (UniqueName: \"kubernetes.io/projected/033b364b-845f-4e4c-b38b-f8e47d81f15e-kube-api-access-4jww2\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:13.757839 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.757757    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/033b364b-845f-4e4c-b38b-f8e47d81f15e-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:13.758055 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:29:13.757856    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Jul 28 02:29:13.758055 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:29:13.757926    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/033b364b-845f-4e4c-b38b-f8e47d81f15e-proxy-tls podName:033b364b-845f-4e4c-b38b-f8e47d81f15e nodeName:}" failed. No retries permitted until 2026-07-28 02:29:14.257905737 +0000 UTC m=+3801.267424140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/033b364b-845f-4e4c-b38b-f8e47d81f15e-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" (UID: "033b364b-845f-4e4c-b38b-f8e47d81f15e") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Jul 28 02:29:13.758178 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.758111    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/033b364b-845f-4e4c-b38b-f8e47d81f15e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:13.758417 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.758394    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/033b364b-845f-4e4c-b38b-f8e47d81f15e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:13.758495 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.758450    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/033b364b-845f-4e4c-b38b-f8e47d81f15e-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:13.767647 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:13.767619    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jww2\" (UniqueName: \"kubernetes.io/projected/033b364b-845f-4e4c-b38b-f8e47d81f15e-kube-api-access-4jww2\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:14.009643 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:14.009558    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98" event={"ID":"b4ffc608-6e63-40a4-8049-e3721b05fcaa","Type":"ContainerDied","Data":"7d20b9a5c939bb201573d82c683a6d8ede82e9268953e677c8bd924078fbc353"}
Jul 28 02:29:14.009643 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:14.009592    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"
Jul 28 02:29:14.009643 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:14.009639    2555 scope.go:117] "RemoveContainer" containerID="4748f67eef7339f1e49bd55e685167f84a4701ad5090f351e527d00fe8b6c630"
Jul 28 02:29:14.017710 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:14.017694    2555 scope.go:117] "RemoveContainer" containerID="e27a0f90286052e4e94bda8cd92ed545b9a8c84a101dff5cdff79da29421401c"
Jul 28 02:29:14.024785 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:14.024748    2555 scope.go:117] "RemoveContainer" containerID="48eed156c197e3ee0d4e769a61b8bd0928c17ac64e004114223c1179e0ebf3f5"
Jul 28 02:29:14.026832 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:14.026809    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"]
Jul 28 02:29:14.030457 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:14.030434    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-npn98"]
Jul 28 02:29:14.261552 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:14.261446    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/033b364b-845f-4e4c-b38b-f8e47d81f15e-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:14.263940 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:14.263915    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/033b364b-845f-4e4c-b38b-f8e47d81f15e-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:14.517420 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:14.517342    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:14.642879 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:14.642855    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"]
Jul 28 02:29:14.645217 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:29:14.645189    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033b364b_845f_4e4c_b38b_f8e47d81f15e.slice/crio-1352ca6ac9602a6a166141decc132c3d3a9f09eb89bd19550c1f040836219db6 WatchSource:0}: Error finding container 1352ca6ac9602a6a166141decc132c3d3a9f09eb89bd19550c1f040836219db6: Status 404 returned error can't find the container with id 1352ca6ac9602a6a166141decc132c3d3a9f09eb89bd19550c1f040836219db6
Jul 28 02:29:15.015545 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:15.015488    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" event={"ID":"033b364b-845f-4e4c-b38b-f8e47d81f15e","Type":"ContainerStarted","Data":"0e5bedbbbf9f698d62e36263e2e8a6b596a64599b43e4736fbb0d25cc36d8922"}
Jul 28 02:29:15.016005 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:15.015582    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" event={"ID":"033b364b-845f-4e4c-b38b-f8e47d81f15e","Type":"ContainerStarted","Data":"1352ca6ac9602a6a166141decc132c3d3a9f09eb89bd19550c1f040836219db6"}
Jul 28 02:29:15.451398 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:15.451319    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4ffc608-6e63-40a4-8049-e3721b05fcaa" path="/var/lib/kubelet/pods/b4ffc608-6e63-40a4-8049-e3721b05fcaa/volumes"
Jul 28 02:29:19.031566 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:19.031504    2555 generic.go:356] "Generic (PLEG): container finished" podID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerID="0e5bedbbbf9f698d62e36263e2e8a6b596a64599b43e4736fbb0d25cc36d8922" exitCode=0
Jul 28 02:29:19.032009 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:19.031586    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" event={"ID":"033b364b-845f-4e4c-b38b-f8e47d81f15e","Type":"ContainerDied","Data":"0e5bedbbbf9f698d62e36263e2e8a6b596a64599b43e4736fbb0d25cc36d8922"}
Jul 28 02:29:20.037343 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:20.037302    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" event={"ID":"033b364b-845f-4e4c-b38b-f8e47d81f15e","Type":"ContainerStarted","Data":"5edd9739e2d277a4d72ade390400c34d15f66623bbf998c38e3a5a6e92f374f1"}
Jul 28 02:29:20.037792 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:20.037364    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" event={"ID":"033b364b-845f-4e4c-b38b-f8e47d81f15e","Type":"ContainerStarted","Data":"dc0528693e3d5a70a071e6c2967bc4bf68500facf2e560546cbaeaa846b81fcd"}
Jul 28 02:29:20.037792 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:20.037568    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:20.037792 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:20.037622    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:20.059414 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:20.059367    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" podStartSLOduration=3.047074239 podStartE2EDuration="7.059352581s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.012278342s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:29:20.058794356 +0000 UTC m=+3807.068312802" watchObservedRunningTime="2026-07-28 02:29:20.059352581 +0000 UTC m=+3807.068871006"
Jul 28 02:29:26.047058 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:26.047031    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:29:56.048138 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:29:56.048096    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" podUID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.54:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 02:30:06.047955 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:06.047917    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" podUID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.54:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 02:30:16.048355 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:16.048273    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" podUID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.54:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 02:30:26.048202 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:26.048157    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" podUID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.54:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 02:30:32.464682 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:32.464650    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:30:33.671374 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:33.671338    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"]
Jul 28 02:30:34.287871 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:34.287833    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" podUID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerName="kserve-container" containerID="cri-o://dc0528693e3d5a70a071e6c2967bc4bf68500facf2e560546cbaeaa846b81fcd" gracePeriod=30
Jul 28 02:30:34.288108 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:34.287876    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" podUID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerName="kube-rbac-proxy" containerID="cri-o://5edd9739e2d277a4d72ade390400c34d15f66623bbf998c38e3a5a6e92f374f1" gracePeriod=30
Jul 28 02:30:35.292908 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:35.292869    2555 generic.go:356] "Generic (PLEG): container finished" podID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerID="5edd9739e2d277a4d72ade390400c34d15f66623bbf998c38e3a5a6e92f374f1" exitCode=2
Jul 28 02:30:35.292908 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:35.292917    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" event={"ID":"033b364b-845f-4e4c-b38b-f8e47d81f15e","Type":"ContainerDied","Data":"5edd9739e2d277a4d72ade390400c34d15f66623bbf998c38e3a5a6e92f374f1"}
Jul 28 02:30:36.042712 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:36.042668    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" podUID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Jul 28 02:30:39.119581 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.119551    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:30:39.185720 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.185680    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/033b364b-845f-4e4c-b38b-f8e47d81f15e-cabundle-cert\") pod \"033b364b-845f-4e4c-b38b-f8e47d81f15e\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") "
Jul 28 02:30:39.185923 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.185731    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/033b364b-845f-4e4c-b38b-f8e47d81f15e-kserve-provision-location\") pod \"033b364b-845f-4e4c-b38b-f8e47d81f15e\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") "
Jul 28 02:30:39.185923 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.185770    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jww2\" (UniqueName: \"kubernetes.io/projected/033b364b-845f-4e4c-b38b-f8e47d81f15e-kube-api-access-4jww2\") pod \"033b364b-845f-4e4c-b38b-f8e47d81f15e\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") "
Jul 28 02:30:39.185923 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.185820    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/033b364b-845f-4e4c-b38b-f8e47d81f15e-kube-rbac-proxy-sar-config\") pod \"033b364b-845f-4e4c-b38b-f8e47d81f15e\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") "
Jul 28 02:30:39.185923 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.185845    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/033b364b-845f-4e4c-b38b-f8e47d81f15e-proxy-tls\") pod \"033b364b-845f-4e4c-b38b-f8e47d81f15e\" (UID: \"033b364b-845f-4e4c-b38b-f8e47d81f15e\") "
Jul 28 02:30:39.186155 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.186125    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033b364b-845f-4e4c-b38b-f8e47d81f15e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "033b364b-845f-4e4c-b38b-f8e47d81f15e" (UID: "033b364b-845f-4e4c-b38b-f8e47d81f15e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:30:39.186203 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.186147    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033b364b-845f-4e4c-b38b-f8e47d81f15e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "033b364b-845f-4e4c-b38b-f8e47d81f15e" (UID: "033b364b-845f-4e4c-b38b-f8e47d81f15e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:30:39.186203 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.186163    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033b364b-845f-4e4c-b38b-f8e47d81f15e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "033b364b-845f-4e4c-b38b-f8e47d81f15e" (UID: "033b364b-845f-4e4c-b38b-f8e47d81f15e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:30:39.188139 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.188115    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033b364b-845f-4e4c-b38b-f8e47d81f15e-kube-api-access-4jww2" (OuterVolumeSpecName: "kube-api-access-4jww2") pod "033b364b-845f-4e4c-b38b-f8e47d81f15e" (UID: "033b364b-845f-4e4c-b38b-f8e47d81f15e"). InnerVolumeSpecName "kube-api-access-4jww2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:30:39.188220 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.188122    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033b364b-845f-4e4c-b38b-f8e47d81f15e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "033b364b-845f-4e4c-b38b-f8e47d81f15e" (UID: "033b364b-845f-4e4c-b38b-f8e47d81f15e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:30:39.286833 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.286800    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jww2\" (UniqueName: \"kubernetes.io/projected/033b364b-845f-4e4c-b38b-f8e47d81f15e-kube-api-access-4jww2\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:30:39.286833 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.286831    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/033b364b-845f-4e4c-b38b-f8e47d81f15e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:30:39.287029 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.286842    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/033b364b-845f-4e4c-b38b-f8e47d81f15e-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:30:39.287029 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.286853    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/033b364b-845f-4e4c-b38b-f8e47d81f15e-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:30:39.287029 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.286860    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/033b364b-845f-4e4c-b38b-f8e47d81f15e-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:30:39.308489 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.308456    2555 generic.go:356] "Generic (PLEG): container finished" podID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerID="dc0528693e3d5a70a071e6c2967bc4bf68500facf2e560546cbaeaa846b81fcd" exitCode=0
Jul 28 02:30:39.308651 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.308492    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" event={"ID":"033b364b-845f-4e4c-b38b-f8e47d81f15e","Type":"ContainerDied","Data":"dc0528693e3d5a70a071e6c2967bc4bf68500facf2e560546cbaeaa846b81fcd"}
Jul 28 02:30:39.308651 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.308538    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"
Jul 28 02:30:39.308651 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.308569    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82" event={"ID":"033b364b-845f-4e4c-b38b-f8e47d81f15e","Type":"ContainerDied","Data":"1352ca6ac9602a6a166141decc132c3d3a9f09eb89bd19550c1f040836219db6"}
Jul 28 02:30:39.308651 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.308598    2555 scope.go:117] "RemoveContainer" containerID="5edd9739e2d277a4d72ade390400c34d15f66623bbf998c38e3a5a6e92f374f1"
Jul 28 02:30:39.317550 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.317508    2555 scope.go:117] "RemoveContainer" containerID="dc0528693e3d5a70a071e6c2967bc4bf68500facf2e560546cbaeaa846b81fcd"
Jul 28 02:30:39.324613 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.324594    2555 scope.go:117] "RemoveContainer" containerID="0e5bedbbbf9f698d62e36263e2e8a6b596a64599b43e4736fbb0d25cc36d8922"
Jul 28 02:30:39.331287 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.331262    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"]
Jul 28 02:30:39.332242 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.332228    2555 scope.go:117] "RemoveContainer" containerID="5edd9739e2d277a4d72ade390400c34d15f66623bbf998c38e3a5a6e92f374f1"
Jul 28 02:30:39.332513 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:30:39.332492    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5edd9739e2d277a4d72ade390400c34d15f66623bbf998c38e3a5a6e92f374f1\": container with ID starting with 5edd9739e2d277a4d72ade390400c34d15f66623bbf998c38e3a5a6e92f374f1 not found: ID does not exist" containerID="5edd9739e2d277a4d72ade390400c34d15f66623bbf998c38e3a5a6e92f374f1"
Jul 28 02:30:39.332698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.332542    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5edd9739e2d277a4d72ade390400c34d15f66623bbf998c38e3a5a6e92f374f1"} err="failed to get container status \"5edd9739e2d277a4d72ade390400c34d15f66623bbf998c38e3a5a6e92f374f1\": rpc error: code = NotFound desc = could not find container \"5edd9739e2d277a4d72ade390400c34d15f66623bbf998c38e3a5a6e92f374f1\": container with ID starting with 5edd9739e2d277a4d72ade390400c34d15f66623bbf998c38e3a5a6e92f374f1 not found: ID does not exist"
Jul 28 02:30:39.332698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.332563    2555 scope.go:117] "RemoveContainer" containerID="dc0528693e3d5a70a071e6c2967bc4bf68500facf2e560546cbaeaa846b81fcd"
Jul 28 02:30:39.332885 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:30:39.332855    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0528693e3d5a70a071e6c2967bc4bf68500facf2e560546cbaeaa846b81fcd\": container with ID starting with dc0528693e3d5a70a071e6c2967bc4bf68500facf2e560546cbaeaa846b81fcd not found: ID does not exist" containerID="dc0528693e3d5a70a071e6c2967bc4bf68500facf2e560546cbaeaa846b81fcd"
Jul 28 02:30:39.333072 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.332894    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0528693e3d5a70a071e6c2967bc4bf68500facf2e560546cbaeaa846b81fcd"} err="failed to get container status \"dc0528693e3d5a70a071e6c2967bc4bf68500facf2e560546cbaeaa846b81fcd\": rpc error: code = NotFound desc = could not find container \"dc0528693e3d5a70a071e6c2967bc4bf68500facf2e560546cbaeaa846b81fcd\": container with ID starting with dc0528693e3d5a70a071e6c2967bc4bf68500facf2e560546cbaeaa846b81fcd not found: ID does not exist"
Jul 28 02:30:39.333072 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.332918    2555 scope.go:117] "RemoveContainer" containerID="0e5bedbbbf9f698d62e36263e2e8a6b596a64599b43e4736fbb0d25cc36d8922"
Jul 28 02:30:39.333264 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:30:39.333208    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5bedbbbf9f698d62e36263e2e8a6b596a64599b43e4736fbb0d25cc36d8922\": container with ID starting with 0e5bedbbbf9f698d62e36263e2e8a6b596a64599b43e4736fbb0d25cc36d8922 not found: ID does not exist" containerID="0e5bedbbbf9f698d62e36263e2e8a6b596a64599b43e4736fbb0d25cc36d8922"
Jul 28 02:30:39.333264 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.333239    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5bedbbbf9f698d62e36263e2e8a6b596a64599b43e4736fbb0d25cc36d8922"} err="failed to get container status \"0e5bedbbbf9f698d62e36263e2e8a6b596a64599b43e4736fbb0d25cc36d8922\": rpc error: code = NotFound desc = could not find container \"0e5bedbbbf9f698d62e36263e2e8a6b596a64599b43e4736fbb0d25cc36d8922\": container with ID starting with 0e5bedbbbf9f698d62e36263e2e8a6b596a64599b43e4736fbb0d25cc36d8922 not found: ID does not exist"
Jul 28 02:30:39.334414 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.334396    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dsg82"]
Jul 28 02:30:39.451549 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:39.451497    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="033b364b-845f-4e4c-b38b-f8e47d81f15e" path="/var/lib/kubelet/pods/033b364b-845f-4e4c-b38b-f8e47d81f15e/volumes"
Jul 28 02:30:46.006492 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.006458    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"]
Jul 28 02:30:46.006929 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.006726    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerName="kube-rbac-proxy"
Jul 28 02:30:46.006929 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.006738    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerName="kube-rbac-proxy"
Jul 28 02:30:46.006929 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.006752    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerName="storage-initializer"
Jul 28 02:30:46.006929 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.006757    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerName="storage-initializer"
Jul 28 02:30:46.006929 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.006767    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerName="kserve-container"
Jul 28 02:30:46.006929 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.006772    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerName="kserve-container"
Jul 28 02:30:46.006929 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.006816    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerName="kube-rbac-proxy"
Jul 28 02:30:46.006929 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.006825    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="033b364b-845f-4e4c-b38b-f8e47d81f15e" containerName="kserve-container"
Jul 28 02:30:46.009541 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.009510    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.012721 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.012701    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 28 02:30:46.012721 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.012713    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:30:46.012985 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.012967    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:30:46.013064 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.013024    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:30:46.013398 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.013382    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:30:46.013457 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.013403    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 28 02:30:46.013457 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.013417    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:30:46.021435 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.021413    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"]
Jul 28 02:30:46.032547 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.032506    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-l8njs\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.032692 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.032604    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-l8njs\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.032692 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.032661    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-l8njs\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.032812 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.032756    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp8t5\" (UniqueName: \"kubernetes.io/projected/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kube-api-access-jp8t5\") pod \"isvc-sklearn-predictor-5f4d89cc44-l8njs\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.032915 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.032818    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-l8njs\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.133182 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.133143    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-l8njs\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.133182 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.133181    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-l8njs\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.133410 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.133308    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp8t5\" (UniqueName: \"kubernetes.io/projected/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kube-api-access-jp8t5\") pod \"isvc-sklearn-predictor-5f4d89cc44-l8njs\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.133410 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.133371    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-l8njs\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.133410 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.133401    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-l8njs\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.133588 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:30:46.133497    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Jul 28 02:30:46.133631 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:30:46.133600    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-proxy-tls podName:e79d9bdc-c6cb-4030-ba27-f20b88ce81b3 nodeName:}" failed. No retries permitted until 2026-07-28 02:30:46.633580897 +0000 UTC m=+3893.643099303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-proxy-tls") pod "isvc-sklearn-predictor-5f4d89cc44-l8njs" (UID: "e79d9bdc-c6cb-4030-ba27-f20b88ce81b3") : secret "isvc-sklearn-predictor-serving-cert" not found
Jul 28 02:30:46.133686 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.133627    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-l8njs\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.134041 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.134021    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-l8njs\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.134081 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.134021    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-l8njs\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.142472 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.142448    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp8t5\" (UniqueName: \"kubernetes.io/projected/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kube-api-access-jp8t5\") pod \"isvc-sklearn-predictor-5f4d89cc44-l8njs\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.637845 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.637802    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-l8njs\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.640357 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.640334    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-l8njs\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:46.918901 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:46.918810    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:47.053933 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:47.053899    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"]
Jul 28 02:30:47.057047 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:30:47.057019    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79d9bdc_c6cb_4030_ba27_f20b88ce81b3.slice/crio-d7d4d6fce247637dbe97a7b880d093aee0181dc965b9450320112af9b60299a8 WatchSource:0}: Error finding container d7d4d6fce247637dbe97a7b880d093aee0181dc965b9450320112af9b60299a8: Status 404 returned error can't find the container with id d7d4d6fce247637dbe97a7b880d093aee0181dc965b9450320112af9b60299a8
Jul 28 02:30:47.340947 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:47.340898    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" event={"ID":"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3","Type":"ContainerStarted","Data":"757fca7267393a95f015796c407b69a7ebd6f18b0dc127e5ae34251fd8e1a2c2"}
Jul 28 02:30:47.341122 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:47.340979    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" event={"ID":"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3","Type":"ContainerStarted","Data":"d7d4d6fce247637dbe97a7b880d093aee0181dc965b9450320112af9b60299a8"}
Jul 28 02:30:51.356032 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:51.355996    2555 generic.go:356] "Generic (PLEG): container finished" podID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerID="757fca7267393a95f015796c407b69a7ebd6f18b0dc127e5ae34251fd8e1a2c2" exitCode=0
Jul 28 02:30:51.356491 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:51.356077    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" event={"ID":"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3","Type":"ContainerDied","Data":"757fca7267393a95f015796c407b69a7ebd6f18b0dc127e5ae34251fd8e1a2c2"}
Jul 28 02:30:52.362542 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:52.362488    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" event={"ID":"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3","Type":"ContainerStarted","Data":"94014cfe35950d423a8a69280baa49b650fe581701a8792010b491e3879663bd"}
Jul 28 02:30:52.363024 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:52.362584    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" event={"ID":"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3","Type":"ContainerStarted","Data":"61bad91af2bb3f8617b912c12dec474d7780aa23d1092251c68d5cbdd328acd0"}
Jul 28 02:30:52.363024 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:52.362805    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:52.382461 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:52.382408    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" podStartSLOduration=3.7996454760000002 podStartE2EDuration="7.382390403s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.582744927s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:30:52.381335891 +0000 UTC m=+3899.390854344" watchObservedRunningTime="2026-07-28 02:30:52.382390403 +0000 UTC m=+3899.391908827"
Jul 28 02:30:53.366582 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:53.366550    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:53.367593 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:53.367560    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 02:30:54.369822 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:54.369782    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 02:30:59.374065 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:59.374035    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:30:59.374566 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:30:59.374513    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 02:31:09.375021 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:31:09.374980    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 02:31:19.374779 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:31:19.374741    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 02:31:29.375252 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:31:29.375211    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 02:31:39.375273 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:31:39.375231    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 02:31:49.375047 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:31:49.374961    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 02:31:59.375393 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:31:59.375352    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 02:32:08.447705 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:08.447670    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:32:16.133081 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:16.133045    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"]
Jul 28 02:32:16.133489 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:16.133318    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kserve-container" containerID="cri-o://61bad91af2bb3f8617b912c12dec474d7780aa23d1092251c68d5cbdd328acd0" gracePeriod=30
Jul 28 02:32:16.133489 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:16.133380    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kube-rbac-proxy" containerID="cri-o://94014cfe35950d423a8a69280baa49b650fe581701a8792010b491e3879663bd" gracePeriod=30
Jul 28 02:32:16.647362 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:16.647321    2555 generic.go:356] "Generic (PLEG): container finished" podID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerID="94014cfe35950d423a8a69280baa49b650fe581701a8792010b491e3879663bd" exitCode=2
Jul 28 02:32:16.647548 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:16.647393    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" event={"ID":"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3","Type":"ContainerDied","Data":"94014cfe35950d423a8a69280baa49b650fe581701a8792010b491e3879663bd"}
Jul 28 02:32:18.447685 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:18.447646    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 02:32:19.370926 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:19.370887    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Jul 28 02:32:20.575405 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.575383    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:32:20.622215 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.622144    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-proxy-tls\") pod \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") "
Jul 28 02:32:20.622215 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.622202    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-cabundle-cert\") pod \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") "
Jul 28 02:32:20.622397 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.622251    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kube-rbac-proxy-sar-config\") pod \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") "
Jul 28 02:32:20.622397 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.622276    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp8t5\" (UniqueName: \"kubernetes.io/projected/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kube-api-access-jp8t5\") pod \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") "
Jul 28 02:32:20.622397 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.622308    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kserve-provision-location\") pod \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\" (UID: \"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3\") "
Jul 28 02:32:20.622579 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.622505    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" (UID: "e79d9bdc-c6cb-4030-ba27-f20b88ce81b3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:32:20.622696 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.622669    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" (UID: "e79d9bdc-c6cb-4030-ba27-f20b88ce81b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:32:20.622798 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.622694    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" (UID: "e79d9bdc-c6cb-4030-ba27-f20b88ce81b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:32:20.624354 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.624325    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" (UID: "e79d9bdc-c6cb-4030-ba27-f20b88ce81b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:32:20.624439 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.624400    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kube-api-access-jp8t5" (OuterVolumeSpecName: "kube-api-access-jp8t5") pod "e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" (UID: "e79d9bdc-c6cb-4030-ba27-f20b88ce81b3"). InnerVolumeSpecName "kube-api-access-jp8t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:32:20.663737 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.663709    2555 generic.go:356] "Generic (PLEG): container finished" podID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerID="61bad91af2bb3f8617b912c12dec474d7780aa23d1092251c68d5cbdd328acd0" exitCode=0
Jul 28 02:32:20.663846 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.663776    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"
Jul 28 02:32:20.663846 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.663798    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" event={"ID":"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3","Type":"ContainerDied","Data":"61bad91af2bb3f8617b912c12dec474d7780aa23d1092251c68d5cbdd328acd0"}
Jul 28 02:32:20.663920 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.663861    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs" event={"ID":"e79d9bdc-c6cb-4030-ba27-f20b88ce81b3","Type":"ContainerDied","Data":"d7d4d6fce247637dbe97a7b880d093aee0181dc965b9450320112af9b60299a8"}
Jul 28 02:32:20.663920 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.663887    2555 scope.go:117] "RemoveContainer" containerID="94014cfe35950d423a8a69280baa49b650fe581701a8792010b491e3879663bd"
Jul 28 02:32:20.672130 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.672116    2555 scope.go:117] "RemoveContainer" containerID="61bad91af2bb3f8617b912c12dec474d7780aa23d1092251c68d5cbdd328acd0"
Jul 28 02:32:20.679023 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.679004    2555 scope.go:117] "RemoveContainer" containerID="757fca7267393a95f015796c407b69a7ebd6f18b0dc127e5ae34251fd8e1a2c2"
Jul 28 02:32:20.685138 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.685115    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"]
Jul 28 02:32:20.686060 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.686038    2555 scope.go:117] "RemoveContainer" containerID="94014cfe35950d423a8a69280baa49b650fe581701a8792010b491e3879663bd"
Jul 28 02:32:20.686336 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:32:20.686315    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94014cfe35950d423a8a69280baa49b650fe581701a8792010b491e3879663bd\": container with ID starting with 94014cfe35950d423a8a69280baa49b650fe581701a8792010b491e3879663bd not found: ID does not exist" containerID="94014cfe35950d423a8a69280baa49b650fe581701a8792010b491e3879663bd"
Jul 28 02:32:20.686406 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.686346    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94014cfe35950d423a8a69280baa49b650fe581701a8792010b491e3879663bd"} err="failed to get container status \"94014cfe35950d423a8a69280baa49b650fe581701a8792010b491e3879663bd\": rpc error: code = NotFound desc = could not find container \"94014cfe35950d423a8a69280baa49b650fe581701a8792010b491e3879663bd\": container with ID starting with 94014cfe35950d423a8a69280baa49b650fe581701a8792010b491e3879663bd not found: ID does not exist"
Jul 28 02:32:20.686406 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.686372    2555 scope.go:117] "RemoveContainer" containerID="61bad91af2bb3f8617b912c12dec474d7780aa23d1092251c68d5cbdd328acd0"
Jul 28 02:32:20.686736 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:32:20.686717    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61bad91af2bb3f8617b912c12dec474d7780aa23d1092251c68d5cbdd328acd0\": container with ID starting with 61bad91af2bb3f8617b912c12dec474d7780aa23d1092251c68d5cbdd328acd0 not found: ID does not exist" containerID="61bad91af2bb3f8617b912c12dec474d7780aa23d1092251c68d5cbdd328acd0"
Jul 28 02:32:20.686811 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.686742    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61bad91af2bb3f8617b912c12dec474d7780aa23d1092251c68d5cbdd328acd0"} err="failed to get container status \"61bad91af2bb3f8617b912c12dec474d7780aa23d1092251c68d5cbdd328acd0\": rpc error: code = NotFound desc = could not find container \"61bad91af2bb3f8617b912c12dec474d7780aa23d1092251c68d5cbdd328acd0\": container with ID starting with 61bad91af2bb3f8617b912c12dec474d7780aa23d1092251c68d5cbdd328acd0 not found: ID does not exist"
Jul 28 02:32:20.686811 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.686758    2555 scope.go:117] "RemoveContainer" containerID="757fca7267393a95f015796c407b69a7ebd6f18b0dc127e5ae34251fd8e1a2c2"
Jul 28 02:32:20.687004 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:32:20.686985    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"757fca7267393a95f015796c407b69a7ebd6f18b0dc127e5ae34251fd8e1a2c2\": container with ID starting with 757fca7267393a95f015796c407b69a7ebd6f18b0dc127e5ae34251fd8e1a2c2 not found: ID does not exist" containerID="757fca7267393a95f015796c407b69a7ebd6f18b0dc127e5ae34251fd8e1a2c2"
Jul 28 02:32:20.687059 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.687012    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"757fca7267393a95f015796c407b69a7ebd6f18b0dc127e5ae34251fd8e1a2c2"} err="failed to get container status \"757fca7267393a95f015796c407b69a7ebd6f18b0dc127e5ae34251fd8e1a2c2\": rpc error: code = NotFound desc = could not find container \"757fca7267393a95f015796c407b69a7ebd6f18b0dc127e5ae34251fd8e1a2c2\": container with ID starting with 757fca7267393a95f015796c407b69a7ebd6f18b0dc127e5ae34251fd8e1a2c2 not found: ID does not exist"
Jul 28 02:32:20.687265 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.687248    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-l8njs"]
Jul 28 02:32:20.723618 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.723592    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:32:20.723618 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.723614    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:32:20.723755 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.723624    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp8t5\" (UniqueName: \"kubernetes.io/projected/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kube-api-access-jp8t5\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:32:20.723755 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.723632    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:32:20.723755 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:20.723641    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:32:21.289306 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.289274    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"]
Jul 28 02:32:21.289589 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.289576    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="storage-initializer"
Jul 28 02:32:21.289644 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.289591    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="storage-initializer"
Jul 28 02:32:21.289644 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.289603    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kserve-container"
Jul 28 02:32:21.289644 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.289609    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kserve-container"
Jul 28 02:32:21.289644 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.289618    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kube-rbac-proxy"
Jul 28 02:32:21.289644 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.289624    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kube-rbac-proxy"
Jul 28 02:32:21.289801 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.289680    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kserve-container"
Jul 28 02:32:21.289801 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.289687    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" containerName="kube-rbac-proxy"
Jul 28 02:32:21.297920 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.297869    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.300749 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.300693    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 28 02:32:21.301154 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.301100    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 02:32:21.301299 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.301203    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:32:21.302178 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.301888    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:32:21.302178 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.301908    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:32:21.302178 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.301923    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:32:21.302410 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.302363    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:32:21.303390 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.303372    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"]
Jul 28 02:32:21.328377 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.328348    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-992zn\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.328492 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.328380    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96c7w\" (UniqueName: \"kubernetes.io/projected/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kube-api-access-96c7w\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-992zn\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.328492 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.328415    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-992zn\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.328492 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.328431    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67f945ac-40d9-4f53-bb95-0968e90e2e8f-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-992zn\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.328492 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.328464    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f945ac-40d9-4f53-bb95-0968e90e2e8f-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-992zn\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.429691 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.429665    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-992zn\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.429827 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.429704    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96c7w\" (UniqueName: \"kubernetes.io/projected/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kube-api-access-96c7w\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-992zn\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.429827 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.429757    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-992zn\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.429827 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.429781    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67f945ac-40d9-4f53-bb95-0968e90e2e8f-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-992zn\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.429827 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.429807    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f945ac-40d9-4f53-bb95-0968e90e2e8f-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-992zn\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.430066 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.430046    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-992zn\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.430443 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.430418    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-992zn\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.430603 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.430586    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67f945ac-40d9-4f53-bb95-0968e90e2e8f-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-992zn\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.432349 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.432330    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f945ac-40d9-4f53-bb95-0968e90e2e8f-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-992zn\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.439093 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.439073    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96c7w\" (UniqueName: \"kubernetes.io/projected/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kube-api-access-96c7w\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-992zn\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.451863 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.451838    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e79d9bdc-c6cb-4030-ba27-f20b88ce81b3" path="/var/lib/kubelet/pods/e79d9bdc-c6cb-4030-ba27-f20b88ce81b3/volumes"
Jul 28 02:32:21.608934 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.608850    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:21.741418 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:21.741397    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"]
Jul 28 02:32:21.743979 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:32:21.743937    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f945ac_40d9_4f53_bb95_0968e90e2e8f.slice/crio-d60c0ccfb8f0273bdf8832e2808de9dbf5e2e99dc6f7d862478a2b5e59d4834f WatchSource:0}: Error finding container d60c0ccfb8f0273bdf8832e2808de9dbf5e2e99dc6f7d862478a2b5e59d4834f: Status 404 returned error can't find the container with id d60c0ccfb8f0273bdf8832e2808de9dbf5e2e99dc6f7d862478a2b5e59d4834f
Jul 28 02:32:22.674805 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:22.674766    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn" event={"ID":"67f945ac-40d9-4f53-bb95-0968e90e2e8f","Type":"ContainerStarted","Data":"e87fc0226bbc6e4e817c6511abbbabdb385e4772925f2ae71557f2daeaf28ced"}
Jul 28 02:32:22.675289 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:22.674837    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn" event={"ID":"67f945ac-40d9-4f53-bb95-0968e90e2e8f","Type":"ContainerStarted","Data":"d60c0ccfb8f0273bdf8832e2808de9dbf5e2e99dc6f7d862478a2b5e59d4834f"}
Jul 28 02:32:25.685424 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:25.685337    2555 generic.go:356] "Generic (PLEG): container finished" podID="67f945ac-40d9-4f53-bb95-0968e90e2e8f" containerID="e87fc0226bbc6e4e817c6511abbbabdb385e4772925f2ae71557f2daeaf28ced" exitCode=0
Jul 28 02:32:25.685424 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:25.685375    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn" event={"ID":"67f945ac-40d9-4f53-bb95-0968e90e2e8f","Type":"ContainerDied","Data":"e87fc0226bbc6e4e817c6511abbbabdb385e4772925f2ae71557f2daeaf28ced"}
Jul 28 02:32:26.691662 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:26.691625    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn" event={"ID":"67f945ac-40d9-4f53-bb95-0968e90e2e8f","Type":"ContainerStarted","Data":"8f1dd294b1198fadbe938ef90dcec333acef98bf347234850c69f792470cef25"}
Jul 28 02:32:26.692076 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:26.691718    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn" event={"ID":"67f945ac-40d9-4f53-bb95-0968e90e2e8f","Type":"ContainerStarted","Data":"1b96a0137b7e61bd72753aadc28cc7465c438a46b6311c2edbb6be15b16f0e4c"}
Jul 28 02:32:26.692076 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:26.691914    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:26.713586 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:26.713499    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn" podStartSLOduration=2.171457405 podStartE2EDuration="5.713484623s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.542027218s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:32:26.712589446 +0000 UTC m=+3993.722107894" watchObservedRunningTime="2026-07-28 02:32:26.713484623 +0000 UTC m=+3993.723003047"
Jul 28 02:32:27.696190 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:27.696163    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:32:33.705735 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:32:33.705709    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:33:03.758887 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:03.758839    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn" podUID="67f945ac-40d9-4f53-bb95-0968e90e2e8f" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 02:33:13.708499 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:13.708462    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:33:21.350988 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:21.350948    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"]
Jul 28 02:33:21.351390 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:21.351265    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn" podUID="67f945ac-40d9-4f53-bb95-0968e90e2e8f" containerName="kserve-container" containerID="cri-o://1b96a0137b7e61bd72753aadc28cc7465c438a46b6311c2edbb6be15b16f0e4c" gracePeriod=30
Jul 28 02:33:21.351390 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:21.351311    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn" podUID="67f945ac-40d9-4f53-bb95-0968e90e2e8f" containerName="kube-rbac-proxy" containerID="cri-o://8f1dd294b1198fadbe938ef90dcec333acef98bf347234850c69f792470cef25" gracePeriod=30
Jul 28 02:33:21.885109 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:21.885079    2555 generic.go:356] "Generic (PLEG): container finished" podID="67f945ac-40d9-4f53-bb95-0968e90e2e8f" containerID="8f1dd294b1198fadbe938ef90dcec333acef98bf347234850c69f792470cef25" exitCode=2
Jul 28 02:33:21.885269 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:21.885119    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn" event={"ID":"67f945ac-40d9-4f53-bb95-0968e90e2e8f","Type":"ContainerDied","Data":"8f1dd294b1198fadbe938ef90dcec333acef98bf347234850c69f792470cef25"}
Jul 28 02:33:23.700845 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:23.700798    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn" podUID="67f945ac-40d9-4f53-bb95-0968e90e2e8f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Jul 28 02:33:28.292428 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.292405    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:33:28.406096 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.406015    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f945ac-40d9-4f53-bb95-0968e90e2e8f-proxy-tls\") pod \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") "
Jul 28 02:33:28.406096 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.406052    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67f945ac-40d9-4f53-bb95-0968e90e2e8f-cabundle-cert\") pod \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") "
Jul 28 02:33:28.406096 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.406078    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kube-rbac-proxy-sar-config\") pod \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") "
Jul 28 02:33:28.406357 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.406198    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kserve-provision-location\") pod \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") "
Jul 28 02:33:28.406357 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.406245    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96c7w\" (UniqueName: \"kubernetes.io/projected/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kube-api-access-96c7w\") pod \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\" (UID: \"67f945ac-40d9-4f53-bb95-0968e90e2e8f\") "
Jul 28 02:33:28.406456 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.406427    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f945ac-40d9-4f53-bb95-0968e90e2e8f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "67f945ac-40d9-4f53-bb95-0968e90e2e8f" (UID: "67f945ac-40d9-4f53-bb95-0968e90e2e8f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:33:28.406498 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.406432    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "67f945ac-40d9-4f53-bb95-0968e90e2e8f" (UID: "67f945ac-40d9-4f53-bb95-0968e90e2e8f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:33:28.406498 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.406473    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "67f945ac-40d9-4f53-bb95-0968e90e2e8f" (UID: "67f945ac-40d9-4f53-bb95-0968e90e2e8f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:33:28.408286 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.408262    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67f945ac-40d9-4f53-bb95-0968e90e2e8f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "67f945ac-40d9-4f53-bb95-0968e90e2e8f" (UID: "67f945ac-40d9-4f53-bb95-0968e90e2e8f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:33:28.408388 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.408369    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kube-api-access-96c7w" (OuterVolumeSpecName: "kube-api-access-96c7w") pod "67f945ac-40d9-4f53-bb95-0968e90e2e8f" (UID: "67f945ac-40d9-4f53-bb95-0968e90e2e8f"). InnerVolumeSpecName "kube-api-access-96c7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:33:28.506790 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.506762    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f945ac-40d9-4f53-bb95-0968e90e2e8f-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:33:28.506790 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.506786    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67f945ac-40d9-4f53-bb95-0968e90e2e8f-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:33:28.506949 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.506797    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:33:28.506949 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.506806    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:33:28.506949 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.506814    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96c7w\" (UniqueName: \"kubernetes.io/projected/67f945ac-40d9-4f53-bb95-0968e90e2e8f-kube-api-access-96c7w\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:33:28.912299 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.912261    2555 generic.go:356] "Generic (PLEG): container finished" podID="67f945ac-40d9-4f53-bb95-0968e90e2e8f" containerID="1b96a0137b7e61bd72753aadc28cc7465c438a46b6311c2edbb6be15b16f0e4c" exitCode=0
Jul 28 02:33:28.912467 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.912327    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"
Jul 28 02:33:28.912467 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.912343    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn" event={"ID":"67f945ac-40d9-4f53-bb95-0968e90e2e8f","Type":"ContainerDied","Data":"1b96a0137b7e61bd72753aadc28cc7465c438a46b6311c2edbb6be15b16f0e4c"}
Jul 28 02:33:28.912467 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.912405    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn" event={"ID":"67f945ac-40d9-4f53-bb95-0968e90e2e8f","Type":"ContainerDied","Data":"d60c0ccfb8f0273bdf8832e2808de9dbf5e2e99dc6f7d862478a2b5e59d4834f"}
Jul 28 02:33:28.912467 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.912431    2555 scope.go:117] "RemoveContainer" containerID="8f1dd294b1198fadbe938ef90dcec333acef98bf347234850c69f792470cef25"
Jul 28 02:33:28.920898 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.920879    2555 scope.go:117] "RemoveContainer" containerID="1b96a0137b7e61bd72753aadc28cc7465c438a46b6311c2edbb6be15b16f0e4c"
Jul 28 02:33:28.927686 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.927666    2555 scope.go:117] "RemoveContainer" containerID="e87fc0226bbc6e4e817c6511abbbabdb385e4772925f2ae71557f2daeaf28ced"
Jul 28 02:33:28.934039 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.934015    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"]
Jul 28 02:33:28.935301 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.935281    2555 scope.go:117] "RemoveContainer" containerID="8f1dd294b1198fadbe938ef90dcec333acef98bf347234850c69f792470cef25"
Jul 28 02:33:28.935556 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:33:28.935516    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f1dd294b1198fadbe938ef90dcec333acef98bf347234850c69f792470cef25\": container with ID starting with 8f1dd294b1198fadbe938ef90dcec333acef98bf347234850c69f792470cef25 not found: ID does not exist" containerID="8f1dd294b1198fadbe938ef90dcec333acef98bf347234850c69f792470cef25"
Jul 28 02:33:28.935601 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.935567    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f1dd294b1198fadbe938ef90dcec333acef98bf347234850c69f792470cef25"} err="failed to get container status \"8f1dd294b1198fadbe938ef90dcec333acef98bf347234850c69f792470cef25\": rpc error: code = NotFound desc = could not find container \"8f1dd294b1198fadbe938ef90dcec333acef98bf347234850c69f792470cef25\": container with ID starting with 8f1dd294b1198fadbe938ef90dcec333acef98bf347234850c69f792470cef25 not found: ID does not exist"
Jul 28 02:33:28.935601 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.935586    2555 scope.go:117] "RemoveContainer" containerID="1b96a0137b7e61bd72753aadc28cc7465c438a46b6311c2edbb6be15b16f0e4c"
Jul 28 02:33:28.935811 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:33:28.935797    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b96a0137b7e61bd72753aadc28cc7465c438a46b6311c2edbb6be15b16f0e4c\": container with ID starting with 1b96a0137b7e61bd72753aadc28cc7465c438a46b6311c2edbb6be15b16f0e4c not found: ID does not exist" containerID="1b96a0137b7e61bd72753aadc28cc7465c438a46b6311c2edbb6be15b16f0e4c"
Jul 28 02:33:28.935857 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.935816    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b96a0137b7e61bd72753aadc28cc7465c438a46b6311c2edbb6be15b16f0e4c"} err="failed to get container status \"1b96a0137b7e61bd72753aadc28cc7465c438a46b6311c2edbb6be15b16f0e4c\": rpc error: code = NotFound desc = could not find container \"1b96a0137b7e61bd72753aadc28cc7465c438a46b6311c2edbb6be15b16f0e4c\": container with ID starting with 1b96a0137b7e61bd72753aadc28cc7465c438a46b6311c2edbb6be15b16f0e4c not found: ID does not exist"
Jul 28 02:33:28.935857 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.935827    2555 scope.go:117] "RemoveContainer" containerID="e87fc0226bbc6e4e817c6511abbbabdb385e4772925f2ae71557f2daeaf28ced"
Jul 28 02:33:28.936099 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:33:28.936077    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e87fc0226bbc6e4e817c6511abbbabdb385e4772925f2ae71557f2daeaf28ced\": container with ID starting with e87fc0226bbc6e4e817c6511abbbabdb385e4772925f2ae71557f2daeaf28ced not found: ID does not exist" containerID="e87fc0226bbc6e4e817c6511abbbabdb385e4772925f2ae71557f2daeaf28ced"
Jul 28 02:33:28.936168 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.936107    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e87fc0226bbc6e4e817c6511abbbabdb385e4772925f2ae71557f2daeaf28ced"} err="failed to get container status \"e87fc0226bbc6e4e817c6511abbbabdb385e4772925f2ae71557f2daeaf28ced\": rpc error: code = NotFound desc = could not find container \"e87fc0226bbc6e4e817c6511abbbabdb385e4772925f2ae71557f2daeaf28ced\": container with ID starting with e87fc0226bbc6e4e817c6511abbbabdb385e4772925f2ae71557f2daeaf28ced not found: ID does not exist"
Jul 28 02:33:28.937994 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:28.937973    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-992zn"]
Jul 28 02:33:29.451119 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:33:29.451086    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67f945ac-40d9-4f53-bb95-0968e90e2e8f" path="/var/lib/kubelet/pods/67f945ac-40d9-4f53-bb95-0968e90e2e8f/volumes"
Jul 28 02:34:03.667937 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.667902    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"]
Jul 28 02:34:03.668388 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.668144    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f945ac-40d9-4f53-bb95-0968e90e2e8f" containerName="kube-rbac-proxy"
Jul 28 02:34:03.668388 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.668155    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f945ac-40d9-4f53-bb95-0968e90e2e8f" containerName="kube-rbac-proxy"
Jul 28 02:34:03.668388 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.668172    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f945ac-40d9-4f53-bb95-0968e90e2e8f" containerName="storage-initializer"
Jul 28 02:34:03.668388 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.668177    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f945ac-40d9-4f53-bb95-0968e90e2e8f" containerName="storage-initializer"
Jul 28 02:34:03.668388 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.668188    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f945ac-40d9-4f53-bb95-0968e90e2e8f" containerName="kserve-container"
Jul 28 02:34:03.668388 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.668195    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f945ac-40d9-4f53-bb95-0968e90e2e8f" containerName="kserve-container"
Jul 28 02:34:03.668388 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.668237    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f945ac-40d9-4f53-bb95-0968e90e2e8f" containerName="kube-rbac-proxy"
Jul 28 02:34:03.668388 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.668243    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f945ac-40d9-4f53-bb95-0968e90e2e8f" containerName="kserve-container"
Jul 28 02:34:03.672545 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.672509    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:03.675639 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.675613    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 28 02:34:03.675753 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.675728    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:34:03.675819 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.675787    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:34:03.675819 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.675788    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 02:34:03.676722 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.676704    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:34:03.676722 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.676710    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:34:03.676722 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.676718    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:34:03.685122 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.685103    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"]
Jul 28 02:34:03.748810 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.748780    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2076d87-3da4-48fa-a314-2c57129ebcf3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:03.748977 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.748815    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2076d87-3da4-48fa-a314-2c57129ebcf3-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:03.748977 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.748835    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2076d87-3da4-48fa-a314-2c57129ebcf3-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:03.748977 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.748853    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2076d87-3da4-48fa-a314-2c57129ebcf3-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:03.748977 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.748873    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48x8h\" (UniqueName: \"kubernetes.io/projected/d2076d87-3da4-48fa-a314-2c57129ebcf3-kube-api-access-48x8h\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:03.849452 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.849401    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2076d87-3da4-48fa-a314-2c57129ebcf3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:03.849452 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.849455    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2076d87-3da4-48fa-a314-2c57129ebcf3-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:03.849651 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:34:03.849574    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 28 02:34:03.849698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.849570    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2076d87-3da4-48fa-a314-2c57129ebcf3-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:03.849698 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:34:03.849651    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2076d87-3da4-48fa-a314-2c57129ebcf3-proxy-tls podName:d2076d87-3da4-48fa-a314-2c57129ebcf3 nodeName:}" failed. No retries permitted until 2026-07-28 02:34:04.349628999 +0000 UTC m=+4091.359147402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d2076d87-3da4-48fa-a314-2c57129ebcf3-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4" (UID: "d2076d87-3da4-48fa-a314-2c57129ebcf3") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 28 02:34:03.849777 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.849712    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2076d87-3da4-48fa-a314-2c57129ebcf3-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:03.849777 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.849744    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48x8h\" (UniqueName: \"kubernetes.io/projected/d2076d87-3da4-48fa-a314-2c57129ebcf3-kube-api-access-48x8h\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:03.849969 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.849946    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2076d87-3da4-48fa-a314-2c57129ebcf3-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:03.850146 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.850128    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2076d87-3da4-48fa-a314-2c57129ebcf3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:03.850214 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.850199    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2076d87-3da4-48fa-a314-2c57129ebcf3-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:03.858062 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:03.858040    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48x8h\" (UniqueName: \"kubernetes.io/projected/d2076d87-3da4-48fa-a314-2c57129ebcf3-kube-api-access-48x8h\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:04.354829 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:04.354782    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2076d87-3da4-48fa-a314-2c57129ebcf3-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:04.357296 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:04.357277    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2076d87-3da4-48fa-a314-2c57129ebcf3-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:04.582549 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:04.582486    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:04.714481 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:04.714381    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"]
Jul 28 02:34:04.717182 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:34:04.717145    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2076d87_3da4_48fa_a314_2c57129ebcf3.slice/crio-7845b490dfcec0b9398d8a9f7adb0103bc78c9870a3075a21a19411c46568ed8 WatchSource:0}: Error finding container 7845b490dfcec0b9398d8a9f7adb0103bc78c9870a3075a21a19411c46568ed8: Status 404 returned error can't find the container with id 7845b490dfcec0b9398d8a9f7adb0103bc78c9870a3075a21a19411c46568ed8
Jul 28 02:34:05.031868 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:05.031833    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4" event={"ID":"d2076d87-3da4-48fa-a314-2c57129ebcf3","Type":"ContainerStarted","Data":"72dac79e5eb833fadc45066a55502cb8290fae3ba1485751004a90c880289b6e"}
Jul 28 02:34:05.032044 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:05.031888    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4" event={"ID":"d2076d87-3da4-48fa-a314-2c57129ebcf3","Type":"ContainerStarted","Data":"7845b490dfcec0b9398d8a9f7adb0103bc78c9870a3075a21a19411c46568ed8"}
Jul 28 02:34:09.047117 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:09.047079    2555 generic.go:356] "Generic (PLEG): container finished" podID="d2076d87-3da4-48fa-a314-2c57129ebcf3" containerID="72dac79e5eb833fadc45066a55502cb8290fae3ba1485751004a90c880289b6e" exitCode=0
Jul 28 02:34:09.047575 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:09.047150    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4" event={"ID":"d2076d87-3da4-48fa-a314-2c57129ebcf3","Type":"ContainerDied","Data":"72dac79e5eb833fadc45066a55502cb8290fae3ba1485751004a90c880289b6e"}
Jul 28 02:34:10.052506 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:10.052471    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4" event={"ID":"d2076d87-3da4-48fa-a314-2c57129ebcf3","Type":"ContainerStarted","Data":"c2dcc562c30fc608e19c7faae51368e5764734edc45a836ddf591c60d337c898"}
Jul 28 02:34:10.052911 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:10.052556    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4" event={"ID":"d2076d87-3da4-48fa-a314-2c57129ebcf3","Type":"ContainerStarted","Data":"be51afdcf3677ea830cfeccaba848be892fbe7f428cad6dc55e09769cc9b285a"}
Jul 28 02:34:10.052911 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:10.052747    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:10.073321 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:10.073272    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4" podStartSLOduration=3.392705775 podStartE2EDuration="7.073256525s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.68055075s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:34:10.072510501 +0000 UTC m=+4097.082028950" watchObservedRunningTime="2026-07-28 02:34:10.073256525 +0000 UTC m=+4097.082774950"
Jul 28 02:34:11.055631 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:11.055588    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:17.064251 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:17.064223    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:34:47.158959 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:47.158864    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4" podUID="d2076d87-3da4-48fa-a314-2c57129ebcf3" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 02:34:57.067219 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:34:57.067184    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:35:03.723588 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:03.723554    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"]
Jul 28 02:35:03.724065 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:03.723847    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4" podUID="d2076d87-3da4-48fa-a314-2c57129ebcf3" containerName="kserve-container" containerID="cri-o://be51afdcf3677ea830cfeccaba848be892fbe7f428cad6dc55e09769cc9b285a" gracePeriod=30
Jul 28 02:35:03.724065 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:03.723914    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4" podUID="d2076d87-3da4-48fa-a314-2c57129ebcf3" containerName="kube-rbac-proxy" containerID="cri-o://c2dcc562c30fc608e19c7faae51368e5764734edc45a836ddf591c60d337c898" gracePeriod=30
Jul 28 02:35:04.238595 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:04.238554    2555 generic.go:356] "Generic (PLEG): container finished" podID="d2076d87-3da4-48fa-a314-2c57129ebcf3" containerID="c2dcc562c30fc608e19c7faae51368e5764734edc45a836ddf591c60d337c898" exitCode=2
Jul 28 02:35:04.238783 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:04.238627    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4" event={"ID":"d2076d87-3da4-48fa-a314-2c57129ebcf3","Type":"ContainerDied","Data":"c2dcc562c30fc608e19c7faae51368e5764734edc45a836ddf591c60d337c898"}
Jul 28 02:35:07.059748 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:07.059701    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4" podUID="d2076d87-3da4-48fa-a314-2c57129ebcf3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Jul 28 02:35:08.960093 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:08.960059    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"]
Jul 28 02:35:08.963408 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:08.963385    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:08.966745 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:08.966726    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 28 02:35:08.968709 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:08.968690    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 28 02:35:08.990096 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:08.990060    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"]
Jul 28 02:35:09.050223 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.050187    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/544f50c0-2269-4982-9bbc-9e10e1e441ef-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-skwcw\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:09.050399 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.050235    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/544f50c0-2269-4982-9bbc-9e10e1e441ef-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-skwcw\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:09.050399 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.050287    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/544f50c0-2269-4982-9bbc-9e10e1e441ef-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-skwcw\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:09.050399 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.050309    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/544f50c0-2269-4982-9bbc-9e10e1e441ef-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-skwcw\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:09.050399 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.050363    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xscgv\" (UniqueName: \"kubernetes.io/projected/544f50c0-2269-4982-9bbc-9e10e1e441ef-kube-api-access-xscgv\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-skwcw\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:09.150730 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.150693    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xscgv\" (UniqueName: \"kubernetes.io/projected/544f50c0-2269-4982-9bbc-9e10e1e441ef-kube-api-access-xscgv\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-skwcw\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:09.150914 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.150743    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/544f50c0-2269-4982-9bbc-9e10e1e441ef-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-skwcw\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:09.150914 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.150774    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/544f50c0-2269-4982-9bbc-9e10e1e441ef-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-skwcw\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:09.150914 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.150803    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/544f50c0-2269-4982-9bbc-9e10e1e441ef-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-skwcw\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:09.150914 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.150825    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/544f50c0-2269-4982-9bbc-9e10e1e441ef-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-skwcw\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:09.151144 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:35:09.150934    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 28 02:35:09.151144 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:35:09.150995    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/544f50c0-2269-4982-9bbc-9e10e1e441ef-proxy-tls podName:544f50c0-2269-4982-9bbc-9e10e1e441ef nodeName:}" failed. No retries permitted until 2026-07-28 02:35:09.650977044 +0000 UTC m=+4156.660495448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/544f50c0-2269-4982-9bbc-9e10e1e441ef-proxy-tls") pod "isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" (UID: "544f50c0-2269-4982-9bbc-9e10e1e441ef") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 28 02:35:09.151325 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.151308    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/544f50c0-2269-4982-9bbc-9e10e1e441ef-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-skwcw\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:09.151566 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.151548    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/544f50c0-2269-4982-9bbc-9e10e1e441ef-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-skwcw\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:09.151609 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.151559    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/544f50c0-2269-4982-9bbc-9e10e1e441ef-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-skwcw\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:09.160038 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.160014    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xscgv\" (UniqueName: \"kubernetes.io/projected/544f50c0-2269-4982-9bbc-9e10e1e441ef-kube-api-access-xscgv\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-skwcw\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:09.654650 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.654616    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/544f50c0-2269-4982-9bbc-9e10e1e441ef-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-skwcw\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:09.657229 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.657194    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/544f50c0-2269-4982-9bbc-9e10e1e441ef-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-skwcw\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:09.873753 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:09.873713    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:10.004973 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:10.004785    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"]
Jul 28 02:35:10.007577 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:35:10.007535    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544f50c0_2269_4982_9bbc_9e10e1e441ef.slice/crio-bdc319e44a1a1d5d4d88df9013992c6bef80b8a60cdbbf4dd010ad47f9a4b7b1 WatchSource:0}: Error finding container bdc319e44a1a1d5d4d88df9013992c6bef80b8a60cdbbf4dd010ad47f9a4b7b1: Status 404 returned error can't find the container with id bdc319e44a1a1d5d4d88df9013992c6bef80b8a60cdbbf4dd010ad47f9a4b7b1
Jul 28 02:35:10.259287 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:10.259240    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" event={"ID":"544f50c0-2269-4982-9bbc-9e10e1e441ef","Type":"ContainerStarted","Data":"4b1dbdb4d47a2dd953341feedadee72e31f5f5b2d217b3f4b581d259f1088331"}
Jul 28 02:35:10.259494 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:10.259313    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" event={"ID":"544f50c0-2269-4982-9bbc-9e10e1e441ef","Type":"ContainerStarted","Data":"bdc319e44a1a1d5d4d88df9013992c6bef80b8a60cdbbf4dd010ad47f9a4b7b1"}
Jul 28 02:35:11.265385 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.265345    2555 generic.go:356] "Generic (PLEG): container finished" podID="d2076d87-3da4-48fa-a314-2c57129ebcf3" containerID="be51afdcf3677ea830cfeccaba848be892fbe7f428cad6dc55e09769cc9b285a" exitCode=0
Jul 28 02:35:11.265900 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.265431    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4" event={"ID":"d2076d87-3da4-48fa-a314-2c57129ebcf3","Type":"ContainerDied","Data":"be51afdcf3677ea830cfeccaba848be892fbe7f428cad6dc55e09769cc9b285a"}
Jul 28 02:35:11.368246 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.368218    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:35:11.466758 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.466678    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2076d87-3da4-48fa-a314-2c57129ebcf3-kserve-provision-location\") pod \"d2076d87-3da4-48fa-a314-2c57129ebcf3\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") "
Jul 28 02:35:11.466758 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.466725    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2076d87-3da4-48fa-a314-2c57129ebcf3-kube-rbac-proxy-sar-config\") pod \"d2076d87-3da4-48fa-a314-2c57129ebcf3\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") "
Jul 28 02:35:11.466758 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.466758    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48x8h\" (UniqueName: \"kubernetes.io/projected/d2076d87-3da4-48fa-a314-2c57129ebcf3-kube-api-access-48x8h\") pod \"d2076d87-3da4-48fa-a314-2c57129ebcf3\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") "
Jul 28 02:35:11.466997 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.466791    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2076d87-3da4-48fa-a314-2c57129ebcf3-cabundle-cert\") pod \"d2076d87-3da4-48fa-a314-2c57129ebcf3\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") "
Jul 28 02:35:11.466997 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.466847    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2076d87-3da4-48fa-a314-2c57129ebcf3-proxy-tls\") pod \"d2076d87-3da4-48fa-a314-2c57129ebcf3\" (UID: \"d2076d87-3da4-48fa-a314-2c57129ebcf3\") "
Jul 28 02:35:11.467107 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.467092    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2076d87-3da4-48fa-a314-2c57129ebcf3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2076d87-3da4-48fa-a314-2c57129ebcf3" (UID: "d2076d87-3da4-48fa-a314-2c57129ebcf3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:35:11.467196 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.467167    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2076d87-3da4-48fa-a314-2c57129ebcf3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2076d87-3da4-48fa-a314-2c57129ebcf3" (UID: "d2076d87-3da4-48fa-a314-2c57129ebcf3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:35:11.467196 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.467179    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2076d87-3da4-48fa-a314-2c57129ebcf3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2076d87-3da4-48fa-a314-2c57129ebcf3" (UID: "d2076d87-3da4-48fa-a314-2c57129ebcf3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:35:11.469094 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.469058    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2076d87-3da4-48fa-a314-2c57129ebcf3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2076d87-3da4-48fa-a314-2c57129ebcf3" (UID: "d2076d87-3da4-48fa-a314-2c57129ebcf3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:35:11.469094 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.469081    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2076d87-3da4-48fa-a314-2c57129ebcf3-kube-api-access-48x8h" (OuterVolumeSpecName: "kube-api-access-48x8h") pod "d2076d87-3da4-48fa-a314-2c57129ebcf3" (UID: "d2076d87-3da4-48fa-a314-2c57129ebcf3"). InnerVolumeSpecName "kube-api-access-48x8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:35:11.567944 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.567900    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2076d87-3da4-48fa-a314-2c57129ebcf3-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:35:11.567944 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.567936    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2076d87-3da4-48fa-a314-2c57129ebcf3-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:35:11.567944 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.567946    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2076d87-3da4-48fa-a314-2c57129ebcf3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:35:11.567944 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.567955    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48x8h\" (UniqueName: \"kubernetes.io/projected/d2076d87-3da4-48fa-a314-2c57129ebcf3-kube-api-access-48x8h\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:35:11.568226 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:11.567965    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2076d87-3da4-48fa-a314-2c57129ebcf3-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:35:12.270791 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:12.270749    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4" event={"ID":"d2076d87-3da4-48fa-a314-2c57129ebcf3","Type":"ContainerDied","Data":"7845b490dfcec0b9398d8a9f7adb0103bc78c9870a3075a21a19411c46568ed8"}
Jul 28 02:35:12.270791 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:12.270780    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"
Jul 28 02:35:12.271331 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:12.270831    2555 scope.go:117] "RemoveContainer" containerID="c2dcc562c30fc608e19c7faae51368e5764734edc45a836ddf591c60d337c898"
Jul 28 02:35:12.280829 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:12.280809    2555 scope.go:117] "RemoveContainer" containerID="be51afdcf3677ea830cfeccaba848be892fbe7f428cad6dc55e09769cc9b285a"
Jul 28 02:35:12.288743 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:12.288725    2555 scope.go:117] "RemoveContainer" containerID="72dac79e5eb833fadc45066a55502cb8290fae3ba1485751004a90c880289b6e"
Jul 28 02:35:12.299889 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:12.299863    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"]
Jul 28 02:35:12.304709 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:12.304687    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-dq6q4"]
Jul 28 02:35:13.451910 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:13.451875    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2076d87-3da4-48fa-a314-2c57129ebcf3" path="/var/lib/kubelet/pods/d2076d87-3da4-48fa-a314-2c57129ebcf3/volumes"
Jul 28 02:35:14.280506 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:14.280472    2555 generic.go:356] "Generic (PLEG): container finished" podID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerID="4b1dbdb4d47a2dd953341feedadee72e31f5f5b2d217b3f4b581d259f1088331" exitCode=0
Jul 28 02:35:14.280711 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:14.280559    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" event={"ID":"544f50c0-2269-4982-9bbc-9e10e1e441ef","Type":"ContainerDied","Data":"4b1dbdb4d47a2dd953341feedadee72e31f5f5b2d217b3f4b581d259f1088331"}
Jul 28 02:35:15.285389 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:15.285352    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" event={"ID":"544f50c0-2269-4982-9bbc-9e10e1e441ef","Type":"ContainerStarted","Data":"ee9181bf2a32335abc21fdc2112c7cb6ec52cbea9166f292030bc1faaacdd859"}
Jul 28 02:35:15.285790 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:15.285443    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" event={"ID":"544f50c0-2269-4982-9bbc-9e10e1e441ef","Type":"ContainerStarted","Data":"59d784c3391a0072f21fd4f19ffe577b66c110dc73788aa422d8ebe7399e1699"}
Jul 28 02:35:15.285790 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:15.285674    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:15.309063 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:15.309009    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" podStartSLOduration=3.57711071 podStartE2EDuration="7.308994253s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.731883543s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:35:15.307452218 +0000 UTC m=+4162.316970670" watchObservedRunningTime="2026-07-28 02:35:15.308994253 +0000 UTC m=+4162.318512677"
Jul 28 02:35:16.289467 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:16.289438    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:16.290692 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:16.290659    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 02:35:17.292936 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:17.292890    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 02:35:22.298615 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:22.298577    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:35:22.299084 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:22.299064    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 02:35:32.299387 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:32.299342    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 02:35:42.299474 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:42.299432    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 02:35:52.299314 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:35:52.299276    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 02:36:02.299510 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:02.299459    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 02:36:12.299191 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:12.299152    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 02:36:22.299220 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:22.299178    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 02:36:32.300658 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:32.300625    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:36:39.160159 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:39.160127    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"]
Jul 28 02:36:39.160657 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:39.160363    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kserve-container" containerID="cri-o://59d784c3391a0072f21fd4f19ffe577b66c110dc73788aa422d8ebe7399e1699" gracePeriod=30
Jul 28 02:36:39.160657 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:39.160415    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kube-rbac-proxy" containerID="cri-o://ee9181bf2a32335abc21fdc2112c7cb6ec52cbea9166f292030bc1faaacdd859" gracePeriod=30
Jul 28 02:36:39.579249 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:39.579216    2555 generic.go:356] "Generic (PLEG): container finished" podID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerID="ee9181bf2a32335abc21fdc2112c7cb6ec52cbea9166f292030bc1faaacdd859" exitCode=2
Jul 28 02:36:39.579419 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:39.579288    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" event={"ID":"544f50c0-2269-4982-9bbc-9e10e1e441ef","Type":"ContainerDied","Data":"ee9181bf2a32335abc21fdc2112c7cb6ec52cbea9166f292030bc1faaacdd859"}
Jul 28 02:36:42.294009 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:42.293962    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.58:8643/healthz\": dial tcp 10.133.0.58:8643: connect: connection refused"
Jul 28 02:36:42.299333 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:42.299296    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 02:36:43.505457 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.505430    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:36:43.593620 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.593587    2555 generic.go:356] "Generic (PLEG): container finished" podID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerID="59d784c3391a0072f21fd4f19ffe577b66c110dc73788aa422d8ebe7399e1699" exitCode=0
Jul 28 02:36:43.593768 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.593646    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"
Jul 28 02:36:43.593768 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.593647    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" event={"ID":"544f50c0-2269-4982-9bbc-9e10e1e441ef","Type":"ContainerDied","Data":"59d784c3391a0072f21fd4f19ffe577b66c110dc73788aa422d8ebe7399e1699"}
Jul 28 02:36:43.593768 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.593731    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw" event={"ID":"544f50c0-2269-4982-9bbc-9e10e1e441ef","Type":"ContainerDied","Data":"bdc319e44a1a1d5d4d88df9013992c6bef80b8a60cdbbf4dd010ad47f9a4b7b1"}
Jul 28 02:36:43.593768 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.593759    2555 scope.go:117] "RemoveContainer" containerID="ee9181bf2a32335abc21fdc2112c7cb6ec52cbea9166f292030bc1faaacdd859"
Jul 28 02:36:43.601257 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.601240    2555 scope.go:117] "RemoveContainer" containerID="59d784c3391a0072f21fd4f19ffe577b66c110dc73788aa422d8ebe7399e1699"
Jul 28 02:36:43.607973 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.607958    2555 scope.go:117] "RemoveContainer" containerID="4b1dbdb4d47a2dd953341feedadee72e31f5f5b2d217b3f4b581d259f1088331"
Jul 28 02:36:43.614801 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.614786    2555 scope.go:117] "RemoveContainer" containerID="ee9181bf2a32335abc21fdc2112c7cb6ec52cbea9166f292030bc1faaacdd859"
Jul 28 02:36:43.615010 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:36:43.614992    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9181bf2a32335abc21fdc2112c7cb6ec52cbea9166f292030bc1faaacdd859\": container with ID starting with ee9181bf2a32335abc21fdc2112c7cb6ec52cbea9166f292030bc1faaacdd859 not found: ID does not exist" containerID="ee9181bf2a32335abc21fdc2112c7cb6ec52cbea9166f292030bc1faaacdd859"
Jul 28 02:36:43.615059 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.615018    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9181bf2a32335abc21fdc2112c7cb6ec52cbea9166f292030bc1faaacdd859"} err="failed to get container status \"ee9181bf2a32335abc21fdc2112c7cb6ec52cbea9166f292030bc1faaacdd859\": rpc error: code = NotFound desc = could not find container \"ee9181bf2a32335abc21fdc2112c7cb6ec52cbea9166f292030bc1faaacdd859\": container with ID starting with ee9181bf2a32335abc21fdc2112c7cb6ec52cbea9166f292030bc1faaacdd859 not found: ID does not exist"
Jul 28 02:36:43.615059 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.615040    2555 scope.go:117] "RemoveContainer" containerID="59d784c3391a0072f21fd4f19ffe577b66c110dc73788aa422d8ebe7399e1699"
Jul 28 02:36:43.615255 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:36:43.615239    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d784c3391a0072f21fd4f19ffe577b66c110dc73788aa422d8ebe7399e1699\": container with ID starting with 59d784c3391a0072f21fd4f19ffe577b66c110dc73788aa422d8ebe7399e1699 not found: ID does not exist" containerID="59d784c3391a0072f21fd4f19ffe577b66c110dc73788aa422d8ebe7399e1699"
Jul 28 02:36:43.615305 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.615260    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d784c3391a0072f21fd4f19ffe577b66c110dc73788aa422d8ebe7399e1699"} err="failed to get container status \"59d784c3391a0072f21fd4f19ffe577b66c110dc73788aa422d8ebe7399e1699\": rpc error: code = NotFound desc = could not find container \"59d784c3391a0072f21fd4f19ffe577b66c110dc73788aa422d8ebe7399e1699\": container with ID starting with 59d784c3391a0072f21fd4f19ffe577b66c110dc73788aa422d8ebe7399e1699 not found: ID does not exist"
Jul 28 02:36:43.615305 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.615272    2555 scope.go:117] "RemoveContainer" containerID="4b1dbdb4d47a2dd953341feedadee72e31f5f5b2d217b3f4b581d259f1088331"
Jul 28 02:36:43.615467 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:36:43.615448    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1dbdb4d47a2dd953341feedadee72e31f5f5b2d217b3f4b581d259f1088331\": container with ID starting with 4b1dbdb4d47a2dd953341feedadee72e31f5f5b2d217b3f4b581d259f1088331 not found: ID does not exist" containerID="4b1dbdb4d47a2dd953341feedadee72e31f5f5b2d217b3f4b581d259f1088331"
Jul 28 02:36:43.615505 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.615473    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1dbdb4d47a2dd953341feedadee72e31f5f5b2d217b3f4b581d259f1088331"} err="failed to get container status \"4b1dbdb4d47a2dd953341feedadee72e31f5f5b2d217b3f4b581d259f1088331\": rpc error: code = NotFound desc = could not find container \"4b1dbdb4d47a2dd953341feedadee72e31f5f5b2d217b3f4b581d259f1088331\": container with ID starting with 4b1dbdb4d47a2dd953341feedadee72e31f5f5b2d217b3f4b581d259f1088331 not found: ID does not exist"
Jul 28 02:36:43.616670 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.616656    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/544f50c0-2269-4982-9bbc-9e10e1e441ef-kserve-provision-location\") pod \"544f50c0-2269-4982-9bbc-9e10e1e441ef\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") "
Jul 28 02:36:43.616722 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.616684    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/544f50c0-2269-4982-9bbc-9e10e1e441ef-cabundle-cert\") pod \"544f50c0-2269-4982-9bbc-9e10e1e441ef\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") "
Jul 28 02:36:43.616722 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.616703    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xscgv\" (UniqueName: \"kubernetes.io/projected/544f50c0-2269-4982-9bbc-9e10e1e441ef-kube-api-access-xscgv\") pod \"544f50c0-2269-4982-9bbc-9e10e1e441ef\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") "
Jul 28 02:36:43.616828 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.616732    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/544f50c0-2269-4982-9bbc-9e10e1e441ef-proxy-tls\") pod \"544f50c0-2269-4982-9bbc-9e10e1e441ef\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") "
Jul 28 02:36:43.616828 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.616758    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/544f50c0-2269-4982-9bbc-9e10e1e441ef-kube-rbac-proxy-sar-config\") pod \"544f50c0-2269-4982-9bbc-9e10e1e441ef\" (UID: \"544f50c0-2269-4982-9bbc-9e10e1e441ef\") "
Jul 28 02:36:43.616994 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.616969    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/544f50c0-2269-4982-9bbc-9e10e1e441ef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "544f50c0-2269-4982-9bbc-9e10e1e441ef" (UID: "544f50c0-2269-4982-9bbc-9e10e1e441ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:36:43.616994 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.616978    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544f50c0-2269-4982-9bbc-9e10e1e441ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "544f50c0-2269-4982-9bbc-9e10e1e441ef" (UID: "544f50c0-2269-4982-9bbc-9e10e1e441ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:36:43.617185 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.617164    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/544f50c0-2269-4982-9bbc-9e10e1e441ef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "544f50c0-2269-4982-9bbc-9e10e1e441ef" (UID: "544f50c0-2269-4982-9bbc-9e10e1e441ef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:36:43.618751 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.618730    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544f50c0-2269-4982-9bbc-9e10e1e441ef-kube-api-access-xscgv" (OuterVolumeSpecName: "kube-api-access-xscgv") pod "544f50c0-2269-4982-9bbc-9e10e1e441ef" (UID: "544f50c0-2269-4982-9bbc-9e10e1e441ef"). InnerVolumeSpecName "kube-api-access-xscgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:36:43.618853 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.618836    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/544f50c0-2269-4982-9bbc-9e10e1e441ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "544f50c0-2269-4982-9bbc-9e10e1e441ef" (UID: "544f50c0-2269-4982-9bbc-9e10e1e441ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:36:43.717785 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.717759    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/544f50c0-2269-4982-9bbc-9e10e1e441ef-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:36:43.717785 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.717782    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/544f50c0-2269-4982-9bbc-9e10e1e441ef-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:36:43.717908 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.717791    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xscgv\" (UniqueName: \"kubernetes.io/projected/544f50c0-2269-4982-9bbc-9e10e1e441ef-kube-api-access-xscgv\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:36:43.717908 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.717800    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/544f50c0-2269-4982-9bbc-9e10e1e441ef-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:36:43.717908 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.717809    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/544f50c0-2269-4982-9bbc-9e10e1e441ef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:36:43.915832 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.915804    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"]
Jul 28 02:36:43.923135 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:43.923113    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-skwcw"]
Jul 28 02:36:44.290803 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.290771    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"]
Jul 28 02:36:44.291008 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.290996    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kserve-container"
Jul 28 02:36:44.291058 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.291009    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kserve-container"
Jul 28 02:36:44.291058 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.291019    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2076d87-3da4-48fa-a314-2c57129ebcf3" containerName="kube-rbac-proxy"
Jul 28 02:36:44.291058 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.291024    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2076d87-3da4-48fa-a314-2c57129ebcf3" containerName="kube-rbac-proxy"
Jul 28 02:36:44.291058 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.291032    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kube-rbac-proxy"
Jul 28 02:36:44.291058 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.291037    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kube-rbac-proxy"
Jul 28 02:36:44.291058 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.291044    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="storage-initializer"
Jul 28 02:36:44.291058 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.291049    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="storage-initializer"
Jul 28 02:36:44.291058 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.291058    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2076d87-3da4-48fa-a314-2c57129ebcf3" containerName="kserve-container"
Jul 28 02:36:44.291303 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.291063    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2076d87-3da4-48fa-a314-2c57129ebcf3" containerName="kserve-container"
Jul 28 02:36:44.291303 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.291069    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2076d87-3da4-48fa-a314-2c57129ebcf3" containerName="storage-initializer"
Jul 28 02:36:44.291303 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.291074    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2076d87-3da4-48fa-a314-2c57129ebcf3" containerName="storage-initializer"
Jul 28 02:36:44.291303 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.291137    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2076d87-3da4-48fa-a314-2c57129ebcf3" containerName="kube-rbac-proxy"
Jul 28 02:36:44.291303 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.291144    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kube-rbac-proxy"
Jul 28 02:36:44.291303 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.291149    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2076d87-3da4-48fa-a314-2c57129ebcf3" containerName="kserve-container"
Jul 28 02:36:44.291303 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.291155    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" containerName="kserve-container"
Jul 28 02:36:44.295110 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.295088    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.297428 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.297407    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 28 02:36:44.297686 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.297668    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 28 02:36:44.297772 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.297750    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:36:44.297832 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.297818    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:36:44.298040 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.298023    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:36:44.298117 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.298028    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:36:44.298649 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.298633    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:36:44.306407 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.306387    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"]
Jul 28 02:36:44.421759 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.421728    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/401a9635-1de7-4970-b669-34203d0d207c-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.421907 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.421762    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/401a9635-1de7-4970-b669-34203d0d207c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.421907 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.421789    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zxjz\" (UniqueName: \"kubernetes.io/projected/401a9635-1de7-4970-b669-34203d0d207c-kube-api-access-5zxjz\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.421907 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.421842    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401a9635-1de7-4970-b669-34203d0d207c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.421907 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.421875    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/401a9635-1de7-4970-b669-34203d0d207c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.522926 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.522894    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zxjz\" (UniqueName: \"kubernetes.io/projected/401a9635-1de7-4970-b669-34203d0d207c-kube-api-access-5zxjz\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.522926 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.522928    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401a9635-1de7-4970-b669-34203d0d207c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.523385 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.522946    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/401a9635-1de7-4970-b669-34203d0d207c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.523385 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.522990    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/401a9635-1de7-4970-b669-34203d0d207c-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.523385 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.523011    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/401a9635-1de7-4970-b669-34203d0d207c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.523572 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.523409    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/401a9635-1de7-4970-b669-34203d0d207c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.523758 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.523738    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/401a9635-1de7-4970-b669-34203d0d207c-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.523829 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.523782    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/401a9635-1de7-4970-b669-34203d0d207c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.525451 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.525431    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401a9635-1de7-4970-b669-34203d0d207c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.530495 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.530472    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zxjz\" (UniqueName: \"kubernetes.io/projected/401a9635-1de7-4970-b669-34203d0d207c-kube-api-access-5zxjz\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.605179 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.605122    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:44.753361 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:44.753336    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"]
Jul 28 02:36:44.755857 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:36:44.755820    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401a9635_1de7_4970_b669_34203d0d207c.slice/crio-10a8eb62fd78039de1093b433db4744de7a580445cdae0ed58df9d92eed5ceca WatchSource:0}: Error finding container 10a8eb62fd78039de1093b433db4744de7a580445cdae0ed58df9d92eed5ceca: Status 404 returned error can't find the container with id 10a8eb62fd78039de1093b433db4744de7a580445cdae0ed58df9d92eed5ceca
Jul 28 02:36:45.451242 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:45.451205    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="544f50c0-2269-4982-9bbc-9e10e1e441ef" path="/var/lib/kubelet/pods/544f50c0-2269-4982-9bbc-9e10e1e441ef/volumes"
Jul 28 02:36:45.603200 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:45.603165    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" event={"ID":"401a9635-1de7-4970-b669-34203d0d207c","Type":"ContainerStarted","Data":"472b0a2665cf27c9cbc8d3f1316fccbd70ca3f8e786875d2a8208f51ecc233c6"}
Jul 28 02:36:45.603588 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:45.603222    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" event={"ID":"401a9635-1de7-4970-b669-34203d0d207c","Type":"ContainerStarted","Data":"10a8eb62fd78039de1093b433db4744de7a580445cdae0ed58df9d92eed5ceca"}
Jul 28 02:36:48.614707 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:48.614672    2555 generic.go:356] "Generic (PLEG): container finished" podID="401a9635-1de7-4970-b669-34203d0d207c" containerID="472b0a2665cf27c9cbc8d3f1316fccbd70ca3f8e786875d2a8208f51ecc233c6" exitCode=0
Jul 28 02:36:48.615063 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:48.614713    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" event={"ID":"401a9635-1de7-4970-b669-34203d0d207c","Type":"ContainerDied","Data":"472b0a2665cf27c9cbc8d3f1316fccbd70ca3f8e786875d2a8208f51ecc233c6"}
Jul 28 02:36:49.619901 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:49.619869    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" event={"ID":"401a9635-1de7-4970-b669-34203d0d207c","Type":"ContainerStarted","Data":"1ada57ba74640ce542e8e74207517267966b6f1b0e3dfc0414ebaa05c41d7d60"}
Jul 28 02:36:49.620265 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:49.619930    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" event={"ID":"401a9635-1de7-4970-b669-34203d0d207c","Type":"ContainerStarted","Data":"e27f47a611502d3660b47842dbfa9bf6f260b745a2fc10f161144814939b7610"}
Jul 28 02:36:49.620265 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:49.620118    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:49.640130 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:49.640081    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" podStartSLOduration=1.982452259 podStartE2EDuration="5.640067263s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.657615004s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:36:49.639354015 +0000 UTC m=+4256.648872440" watchObservedRunningTime="2026-07-28 02:36:49.640067263 +0000 UTC m=+4256.649585688"
Jul 28 02:36:50.623179 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:50.623148    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:50.624332 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:50.624301    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 02:36:51.625909 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:51.625873    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 02:36:56.630171 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:56.630143    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:36:56.630612 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:36:56.630585    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 02:37:06.631464 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:37:06.631420    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 02:37:16.630644 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:37:16.630596    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 02:37:26.631288 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:37:26.631238    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 02:37:36.630749 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:37:36.630705    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 02:37:46.631048 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:37:46.630956    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 02:37:56.630559 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:37:56.630501    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 02:38:06.631693 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:06.631660    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:38:14.445378 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:14.445344    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"]
Jul 28 02:38:14.445826 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:14.445599    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kserve-container" containerID="cri-o://e27f47a611502d3660b47842dbfa9bf6f260b745a2fc10f161144814939b7610" gracePeriod=30
Jul 28 02:38:14.445826 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:14.445653    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kube-rbac-proxy" containerID="cri-o://1ada57ba74640ce542e8e74207517267966b6f1b0e3dfc0414ebaa05c41d7d60" gracePeriod=30
Jul 28 02:38:14.907691 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:14.907651    2555 generic.go:356] "Generic (PLEG): container finished" podID="401a9635-1de7-4970-b669-34203d0d207c" containerID="1ada57ba74640ce542e8e74207517267966b6f1b0e3dfc0414ebaa05c41d7d60" exitCode=2
Jul 28 02:38:14.907855 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:14.907699    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" event={"ID":"401a9635-1de7-4970-b669-34203d0d207c","Type":"ContainerDied","Data":"1ada57ba74640ce542e8e74207517267966b6f1b0e3dfc0414ebaa05c41d7d60"}
Jul 28 02:38:16.627146 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:16.627090    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.59:8643/healthz\": dial tcp 10.133.0.59:8643: connect: connection refused"
Jul 28 02:38:16.631455 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:16.631423    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 02:38:19.086257 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.086232    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:38:19.195382 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.195281    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401a9635-1de7-4970-b669-34203d0d207c-proxy-tls\") pod \"401a9635-1de7-4970-b669-34203d0d207c\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") "
Jul 28 02:38:19.195382 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.195349    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/401a9635-1de7-4970-b669-34203d0d207c-kube-rbac-proxy-sar-config\") pod \"401a9635-1de7-4970-b669-34203d0d207c\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") "
Jul 28 02:38:19.195662 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.195393    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zxjz\" (UniqueName: \"kubernetes.io/projected/401a9635-1de7-4970-b669-34203d0d207c-kube-api-access-5zxjz\") pod \"401a9635-1de7-4970-b669-34203d0d207c\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") "
Jul 28 02:38:19.195662 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.195419    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/401a9635-1de7-4970-b669-34203d0d207c-cabundle-cert\") pod \"401a9635-1de7-4970-b669-34203d0d207c\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") "
Jul 28 02:38:19.195662 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.195448    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/401a9635-1de7-4970-b669-34203d0d207c-kserve-provision-location\") pod \"401a9635-1de7-4970-b669-34203d0d207c\" (UID: \"401a9635-1de7-4970-b669-34203d0d207c\") "
Jul 28 02:38:19.195859 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.195828    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401a9635-1de7-4970-b669-34203d0d207c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "401a9635-1de7-4970-b669-34203d0d207c" (UID: "401a9635-1de7-4970-b669-34203d0d207c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:38:19.195917 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.195835    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401a9635-1de7-4970-b669-34203d0d207c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "401a9635-1de7-4970-b669-34203d0d207c" (UID: "401a9635-1de7-4970-b669-34203d0d207c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:38:19.195917 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.195892    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401a9635-1de7-4970-b669-34203d0d207c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "401a9635-1de7-4970-b669-34203d0d207c" (UID: "401a9635-1de7-4970-b669-34203d0d207c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:38:19.197573 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.197551    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/401a9635-1de7-4970-b669-34203d0d207c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "401a9635-1de7-4970-b669-34203d0d207c" (UID: "401a9635-1de7-4970-b669-34203d0d207c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:38:19.197733 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.197718    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401a9635-1de7-4970-b669-34203d0d207c-kube-api-access-5zxjz" (OuterVolumeSpecName: "kube-api-access-5zxjz") pod "401a9635-1de7-4970-b669-34203d0d207c" (UID: "401a9635-1de7-4970-b669-34203d0d207c"). InnerVolumeSpecName "kube-api-access-5zxjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:38:19.296772 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.296734    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/401a9635-1de7-4970-b669-34203d0d207c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:38:19.296772 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.296764    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zxjz\" (UniqueName: \"kubernetes.io/projected/401a9635-1de7-4970-b669-34203d0d207c-kube-api-access-5zxjz\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:38:19.296772 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.296776    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/401a9635-1de7-4970-b669-34203d0d207c-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:38:19.297003 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.296785    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/401a9635-1de7-4970-b669-34203d0d207c-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:38:19.297003 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.296799    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401a9635-1de7-4970-b669-34203d0d207c-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:38:19.624681 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.624642    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"]
Jul 28 02:38:19.624889 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.624876    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kube-rbac-proxy"
Jul 28 02:38:19.624889 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.624889    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kube-rbac-proxy"
Jul 28 02:38:19.624993 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.624899    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="storage-initializer"
Jul 28 02:38:19.624993 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.624905    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="storage-initializer"
Jul 28 02:38:19.624993 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.624917    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kserve-container"
Jul 28 02:38:19.624993 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.624924    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kserve-container"
Jul 28 02:38:19.624993 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.624972    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kube-rbac-proxy"
Jul 28 02:38:19.624993 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.624978    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="401a9635-1de7-4970-b669-34203d0d207c" containerName="kserve-container"
Jul 28 02:38:19.627937 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.627917    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.630510 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.630486    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jul 28 02:38:19.630760 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.630745    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jul 28 02:38:19.639861 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.639833    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"]
Jul 28 02:38:19.800053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.800018    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-zczfw\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.800053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.800063    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df37e50e-024d-44fa-9fd8-9e2721a5f26e-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-zczfw\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.800319 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.800085    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df37e50e-024d-44fa-9fd8-9e2721a5f26e-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-zczfw\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.800319 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.800153    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-zczfw\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.800319 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.800180    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbvd7\" (UniqueName: \"kubernetes.io/projected/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kube-api-access-gbvd7\") pod \"isvc-tensorflow-predictor-547dfc5669-zczfw\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.900991 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.900951    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-zczfw\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.900991 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.900996    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df37e50e-024d-44fa-9fd8-9e2721a5f26e-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-zczfw\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.901240 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.901020    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df37e50e-024d-44fa-9fd8-9e2721a5f26e-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-zczfw\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.901240 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.901043    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-zczfw\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.901240 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.901150    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbvd7\" (UniqueName: \"kubernetes.io/projected/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kube-api-access-gbvd7\") pod \"isvc-tensorflow-predictor-547dfc5669-zczfw\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.901618 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.901597    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-zczfw\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.901779 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.901758    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df37e50e-024d-44fa-9fd8-9e2721a5f26e-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-zczfw\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.901871 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.901849    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-zczfw\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.903820 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.903803    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df37e50e-024d-44fa-9fd8-9e2721a5f26e-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-zczfw\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.911172 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.911150    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbvd7\" (UniqueName: \"kubernetes.io/projected/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kube-api-access-gbvd7\") pod \"isvc-tensorflow-predictor-547dfc5669-zczfw\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.926177 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.926140    2555 generic.go:356] "Generic (PLEG): container finished" podID="401a9635-1de7-4970-b669-34203d0d207c" containerID="e27f47a611502d3660b47842dbfa9bf6f260b745a2fc10f161144814939b7610" exitCode=0
Jul 28 02:38:19.926349 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.926181    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" event={"ID":"401a9635-1de7-4970-b669-34203d0d207c","Type":"ContainerDied","Data":"e27f47a611502d3660b47842dbfa9bf6f260b745a2fc10f161144814939b7610"}
Jul 28 02:38:19.926349 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.926213    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"
Jul 28 02:38:19.926349 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.926243    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t" event={"ID":"401a9635-1de7-4970-b669-34203d0d207c","Type":"ContainerDied","Data":"10a8eb62fd78039de1093b433db4744de7a580445cdae0ed58df9d92eed5ceca"}
Jul 28 02:38:19.926349 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.926278    2555 scope.go:117] "RemoveContainer" containerID="1ada57ba74640ce542e8e74207517267966b6f1b0e3dfc0414ebaa05c41d7d60"
Jul 28 02:38:19.936551 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.936506    2555 scope.go:117] "RemoveContainer" containerID="e27f47a611502d3660b47842dbfa9bf6f260b745a2fc10f161144814939b7610"
Jul 28 02:38:19.938630 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.938604    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:19.944473 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.944444    2555 scope.go:117] "RemoveContainer" containerID="472b0a2665cf27c9cbc8d3f1316fccbd70ca3f8e786875d2a8208f51ecc233c6"
Jul 28 02:38:19.945813 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.945792    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"]
Jul 28 02:38:19.950240 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.950214    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-6qp6t"]
Jul 28 02:38:19.963197 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.963176    2555 scope.go:117] "RemoveContainer" containerID="1ada57ba74640ce542e8e74207517267966b6f1b0e3dfc0414ebaa05c41d7d60"
Jul 28 02:38:19.963584 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:38:19.963563    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ada57ba74640ce542e8e74207517267966b6f1b0e3dfc0414ebaa05c41d7d60\": container with ID starting with 1ada57ba74640ce542e8e74207517267966b6f1b0e3dfc0414ebaa05c41d7d60 not found: ID does not exist" containerID="1ada57ba74640ce542e8e74207517267966b6f1b0e3dfc0414ebaa05c41d7d60"
Jul 28 02:38:19.963644 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.963595    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ada57ba74640ce542e8e74207517267966b6f1b0e3dfc0414ebaa05c41d7d60"} err="failed to get container status \"1ada57ba74640ce542e8e74207517267966b6f1b0e3dfc0414ebaa05c41d7d60\": rpc error: code = NotFound desc = could not find container \"1ada57ba74640ce542e8e74207517267966b6f1b0e3dfc0414ebaa05c41d7d60\": container with ID starting with 1ada57ba74640ce542e8e74207517267966b6f1b0e3dfc0414ebaa05c41d7d60 not found: ID does not exist"
Jul 28 02:38:19.963644 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.963616    2555 scope.go:117] "RemoveContainer" containerID="e27f47a611502d3660b47842dbfa9bf6f260b745a2fc10f161144814939b7610"
Jul 28 02:38:19.963904 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:38:19.963880    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e27f47a611502d3660b47842dbfa9bf6f260b745a2fc10f161144814939b7610\": container with ID starting with e27f47a611502d3660b47842dbfa9bf6f260b745a2fc10f161144814939b7610 not found: ID does not exist" containerID="e27f47a611502d3660b47842dbfa9bf6f260b745a2fc10f161144814939b7610"
Jul 28 02:38:19.963960 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.963915    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27f47a611502d3660b47842dbfa9bf6f260b745a2fc10f161144814939b7610"} err="failed to get container status \"e27f47a611502d3660b47842dbfa9bf6f260b745a2fc10f161144814939b7610\": rpc error: code = NotFound desc = could not find container \"e27f47a611502d3660b47842dbfa9bf6f260b745a2fc10f161144814939b7610\": container with ID starting with e27f47a611502d3660b47842dbfa9bf6f260b745a2fc10f161144814939b7610 not found: ID does not exist"
Jul 28 02:38:19.963960 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.963941    2555 scope.go:117] "RemoveContainer" containerID="472b0a2665cf27c9cbc8d3f1316fccbd70ca3f8e786875d2a8208f51ecc233c6"
Jul 28 02:38:19.964255 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:38:19.964226    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"472b0a2665cf27c9cbc8d3f1316fccbd70ca3f8e786875d2a8208f51ecc233c6\": container with ID starting with 472b0a2665cf27c9cbc8d3f1316fccbd70ca3f8e786875d2a8208f51ecc233c6 not found: ID does not exist" containerID="472b0a2665cf27c9cbc8d3f1316fccbd70ca3f8e786875d2a8208f51ecc233c6"
Jul 28 02:38:19.964326 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:19.964262    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"472b0a2665cf27c9cbc8d3f1316fccbd70ca3f8e786875d2a8208f51ecc233c6"} err="failed to get container status \"472b0a2665cf27c9cbc8d3f1316fccbd70ca3f8e786875d2a8208f51ecc233c6\": rpc error: code = NotFound desc = could not find container \"472b0a2665cf27c9cbc8d3f1316fccbd70ca3f8e786875d2a8208f51ecc233c6\": container with ID starting with 472b0a2665cf27c9cbc8d3f1316fccbd70ca3f8e786875d2a8208f51ecc233c6 not found: ID does not exist"
Jul 28 02:38:20.080241 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:20.080156    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"]
Jul 28 02:38:20.083416 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:38:20.083389    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf37e50e_024d_44fa_9fd8_9e2721a5f26e.slice/crio-055d82c625ae0159ccbe41f84602710cd4694f8519a968c760469e59baf42dba WatchSource:0}: Error finding container 055d82c625ae0159ccbe41f84602710cd4694f8519a968c760469e59baf42dba: Status 404 returned error can't find the container with id 055d82c625ae0159ccbe41f84602710cd4694f8519a968c760469e59baf42dba
Jul 28 02:38:20.933041 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:20.933000    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" event={"ID":"df37e50e-024d-44fa-9fd8-9e2721a5f26e","Type":"ContainerStarted","Data":"5569fc54e558a2b8ff5c1b7f09d8c8b4fa10ecb6e89a43bec1b76ae213d38dad"}
Jul 28 02:38:20.933554 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:20.933063    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" event={"ID":"df37e50e-024d-44fa-9fd8-9e2721a5f26e","Type":"ContainerStarted","Data":"055d82c625ae0159ccbe41f84602710cd4694f8519a968c760469e59baf42dba"}
Jul 28 02:38:21.452071 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:21.452032    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="401a9635-1de7-4970-b669-34203d0d207c" path="/var/lib/kubelet/pods/401a9635-1de7-4970-b669-34203d0d207c/volumes"
Jul 28 02:38:24.948661 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:24.948570    2555 generic.go:356] "Generic (PLEG): container finished" podID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerID="5569fc54e558a2b8ff5c1b7f09d8c8b4fa10ecb6e89a43bec1b76ae213d38dad" exitCode=0
Jul 28 02:38:24.948661 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:24.948626    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" event={"ID":"df37e50e-024d-44fa-9fd8-9e2721a5f26e","Type":"ContainerDied","Data":"5569fc54e558a2b8ff5c1b7f09d8c8b4fa10ecb6e89a43bec1b76ae213d38dad"}
Jul 28 02:38:24.949863 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:24.949844    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 02:38:28.968342 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:28.968301    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" event={"ID":"df37e50e-024d-44fa-9fd8-9e2721a5f26e","Type":"ContainerStarted","Data":"6aecc680e94c2bda6eb8a763e6c21bbbba9afad60351cc210ae36bd5ac393757"}
Jul 28 02:38:28.968822 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:28.968366    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" event={"ID":"df37e50e-024d-44fa-9fd8-9e2721a5f26e","Type":"ContainerStarted","Data":"e57e46c13b123a18f6d357b510992801cce28fd385f7f657e5d3d10fceecf830"}
Jul 28 02:38:28.968822 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:28.968649    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:28.968822 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:28.968715    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:28.970277 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:28.970250    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 28 02:38:28.990161 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:28.990104    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" podStartSLOduration=1.556040793 podStartE2EDuration="9.990086868s" totalImagesPullingTime="3.883643178s" totalInitContainerRuntime="4.550402897s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:38:28.988613493 +0000 UTC m=+4355.998131939" watchObservedRunningTime="2026-07-28 02:38:28.990086868 +0000 UTC m=+4355.999605294"
Jul 28 02:38:29.972182 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:29.972140    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 28 02:38:34.978190 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:34.978156    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:38:34.978740 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:34.978709    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 28 02:38:44.979303 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:38:44.979268    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:39:01.437155 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:01.437115    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"]
Jul 28 02:39:01.437759 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:01.437346    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kserve-container" containerID="cri-o://e57e46c13b123a18f6d357b510992801cce28fd385f7f657e5d3d10fceecf830" gracePeriod=30
Jul 28 02:39:01.437759 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:01.437373    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kube-rbac-proxy" containerID="cri-o://6aecc680e94c2bda6eb8a763e6c21bbbba9afad60351cc210ae36bd5ac393757" gracePeriod=30
Jul 28 02:39:02.090728 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:02.090695    2555 generic.go:356] "Generic (PLEG): container finished" podID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerID="6aecc680e94c2bda6eb8a763e6c21bbbba9afad60351cc210ae36bd5ac393757" exitCode=2
Jul 28 02:39:02.090728 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:02.090733    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" event={"ID":"df37e50e-024d-44fa-9fd8-9e2721a5f26e","Type":"ContainerDied","Data":"6aecc680e94c2bda6eb8a763e6c21bbbba9afad60351cc210ae36bd5ac393757"}
Jul 28 02:39:04.972640 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:04.972590    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Jul 28 02:39:06.615730 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.615693    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"]
Jul 28 02:39:06.618215 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.618200    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:06.620860 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.620775    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3f4286d-cae0-4999-aefb-0940f8da5770-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-76jzr\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:06.620860 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.620805    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jul 28 02:39:06.621082 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.620784    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jul 28 02:39:06.621082 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.620892    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhpqv\" (UniqueName: \"kubernetes.io/projected/e3f4286d-cae0-4999-aefb-0940f8da5770-kube-api-access-dhpqv\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-76jzr\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:06.621082 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.620948    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3f4286d-cae0-4999-aefb-0940f8da5770-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-76jzr\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:06.621082 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.620974    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3f4286d-cae0-4999-aefb-0940f8da5770-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-76jzr\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:06.621082 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.621029    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3f4286d-cae0-4999-aefb-0940f8da5770-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-76jzr\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:06.629428 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.629406    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"]
Jul 28 02:39:06.721696 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.721656    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3f4286d-cae0-4999-aefb-0940f8da5770-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-76jzr\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:06.721906 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.721704    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3f4286d-cae0-4999-aefb-0940f8da5770-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-76jzr\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:06.721906 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.721743    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhpqv\" (UniqueName: \"kubernetes.io/projected/e3f4286d-cae0-4999-aefb-0940f8da5770-kube-api-access-dhpqv\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-76jzr\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:06.721906 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.721781    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3f4286d-cae0-4999-aefb-0940f8da5770-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-76jzr\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:06.721906 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.721804    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3f4286d-cae0-4999-aefb-0940f8da5770-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-76jzr\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:06.722135 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:39:06.721927    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Jul 28 02:39:06.722135 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:39:06.722033    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f4286d-cae0-4999-aefb-0940f8da5770-proxy-tls podName:e3f4286d-cae0-4999-aefb-0940f8da5770 nodeName:}" failed. No retries permitted until 2026-07-28 02:39:07.222015087 +0000 UTC m=+4394.231533491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e3f4286d-cae0-4999-aefb-0940f8da5770-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" (UID: "e3f4286d-cae0-4999-aefb-0940f8da5770") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Jul 28 02:39:06.722250 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.722225    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3f4286d-cae0-4999-aefb-0940f8da5770-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-76jzr\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:06.722437 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.722416    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3f4286d-cae0-4999-aefb-0940f8da5770-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-76jzr\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:06.722478 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.722442    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3f4286d-cae0-4999-aefb-0940f8da5770-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-76jzr\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:06.731481 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:06.731458    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhpqv\" (UniqueName: \"kubernetes.io/projected/e3f4286d-cae0-4999-aefb-0940f8da5770-kube-api-access-dhpqv\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-76jzr\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:07.225346 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:07.225302    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3f4286d-cae0-4999-aefb-0940f8da5770-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-76jzr\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:07.227991 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:07.227963    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3f4286d-cae0-4999-aefb-0940f8da5770-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-76jzr\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:07.228127 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:07.228100    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:07.361427 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:07.361300    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"]
Jul 28 02:39:07.363987 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:39:07.363949    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f4286d_cae0_4999_aefb_0940f8da5770.slice/crio-849297625bd076c52f752099169df064664bf8fb7f6cef8e8bcec8341f08c12e WatchSource:0}: Error finding container 849297625bd076c52f752099169df064664bf8fb7f6cef8e8bcec8341f08c12e: Status 404 returned error can't find the container with id 849297625bd076c52f752099169df064664bf8fb7f6cef8e8bcec8341f08c12e
Jul 28 02:39:08.112502 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:08.112466    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" event={"ID":"e3f4286d-cae0-4999-aefb-0940f8da5770","Type":"ContainerStarted","Data":"b366310b23e7adbc2347b4cb274075dcc78394cbd0e08fdc8202d21fc4ea021b"}
Jul 28 02:39:08.112907 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:08.112539    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" event={"ID":"e3f4286d-cae0-4999-aefb-0940f8da5770","Type":"ContainerStarted","Data":"849297625bd076c52f752099169df064664bf8fb7f6cef8e8bcec8341f08c12e"}
Jul 28 02:39:09.972414 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:09.972370    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Jul 28 02:39:12.128726 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:12.128695    2555 generic.go:356] "Generic (PLEG): container finished" podID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerID="b366310b23e7adbc2347b4cb274075dcc78394cbd0e08fdc8202d21fc4ea021b" exitCode=0
Jul 28 02:39:12.129101 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:12.128756    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" event={"ID":"e3f4286d-cae0-4999-aefb-0940f8da5770","Type":"ContainerDied","Data":"b366310b23e7adbc2347b4cb274075dcc78394cbd0e08fdc8202d21fc4ea021b"}
Jul 28 02:39:13.134136 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:13.134095    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" event={"ID":"e3f4286d-cae0-4999-aefb-0940f8da5770","Type":"ContainerStarted","Data":"87d095211eece0070f638a9ca72a9e3a4fc7f3646d8c735578a69722ea0e62e5"}
Jul 28 02:39:13.134541 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:13.134167    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" event={"ID":"e3f4286d-cae0-4999-aefb-0940f8da5770","Type":"ContainerStarted","Data":"20375b0b2b03aa7850f193a3999438260b2e08f25f5099cbe91f9cb2a2887117"}
Jul 28 02:39:13.134541 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:13.134361    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:13.155856 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:13.155801    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" podStartSLOduration=2.920149346 podStartE2EDuration="7.155786727s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.235637381s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:39:13.155563414 +0000 UTC m=+4400.165081858" watchObservedRunningTime="2026-07-28 02:39:13.155786727 +0000 UTC m=+4400.165305151"
Jul 28 02:39:14.137724 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:14.137687    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:14.138811 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:14.138785    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 28 02:39:14.973323 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:14.973279    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Jul 28 02:39:14.973502 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:14.973414    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:39:15.140511 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:15.140471    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 28 02:39:19.973260 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:19.973213    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Jul 28 02:39:20.145759 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:20.145730    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:20.146302 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:20.146269    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 28 02:39:24.972669 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:24.972621    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Jul 28 02:39:29.972376 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:29.972328    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Jul 28 02:39:30.147350 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:30.147323    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:39:32.093400 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.093377    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:39:32.204477 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.204396    2555 generic.go:356] "Generic (PLEG): container finished" podID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerID="e57e46c13b123a18f6d357b510992801cce28fd385f7f657e5d3d10fceecf830" exitCode=137
Jul 28 02:39:32.204477 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.204462    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"
Jul 28 02:39:32.204685 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.204486    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" event={"ID":"df37e50e-024d-44fa-9fd8-9e2721a5f26e","Type":"ContainerDied","Data":"e57e46c13b123a18f6d357b510992801cce28fd385f7f657e5d3d10fceecf830"}
Jul 28 02:39:32.204685 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.204567    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw" event={"ID":"df37e50e-024d-44fa-9fd8-9e2721a5f26e","Type":"ContainerDied","Data":"055d82c625ae0159ccbe41f84602710cd4694f8519a968c760469e59baf42dba"}
Jul 28 02:39:32.204685 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.204599    2555 scope.go:117] "RemoveContainer" containerID="6aecc680e94c2bda6eb8a763e6c21bbbba9afad60351cc210ae36bd5ac393757"
Jul 28 02:39:32.208185 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.208160    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df37e50e-024d-44fa-9fd8-9e2721a5f26e-cabundle-cert\") pod \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") "
Jul 28 02:39:32.208307 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.208200    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kube-rbac-proxy-sar-config\") pod \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") "
Jul 28 02:39:32.208307 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.208244    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df37e50e-024d-44fa-9fd8-9e2721a5f26e-proxy-tls\") pod \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") "
Jul 28 02:39:32.208307 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.208301    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbvd7\" (UniqueName: \"kubernetes.io/projected/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kube-api-access-gbvd7\") pod \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") "
Jul 28 02:39:32.208477 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.208349    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kserve-provision-location\") pod \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\" (UID: \"df37e50e-024d-44fa-9fd8-9e2721a5f26e\") "
Jul 28 02:39:32.208617 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.208570    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df37e50e-024d-44fa-9fd8-9e2721a5f26e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "df37e50e-024d-44fa-9fd8-9e2721a5f26e" (UID: "df37e50e-024d-44fa-9fd8-9e2721a5f26e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:39:32.208617 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.208604    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df37e50e-024d-44fa-9fd8-9e2721a5f26e" (UID: "df37e50e-024d-44fa-9fd8-9e2721a5f26e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:39:32.210758 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.210729    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kube-api-access-gbvd7" (OuterVolumeSpecName: "kube-api-access-gbvd7") pod "df37e50e-024d-44fa-9fd8-9e2721a5f26e" (UID: "df37e50e-024d-44fa-9fd8-9e2721a5f26e"). InnerVolumeSpecName "kube-api-access-gbvd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:39:32.210913 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.210883    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df37e50e-024d-44fa-9fd8-9e2721a5f26e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df37e50e-024d-44fa-9fd8-9e2721a5f26e" (UID: "df37e50e-024d-44fa-9fd8-9e2721a5f26e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:39:32.213183 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.213154    2555 scope.go:117] "RemoveContainer" containerID="e57e46c13b123a18f6d357b510992801cce28fd385f7f657e5d3d10fceecf830"
Jul 28 02:39:32.218074 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.218049    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df37e50e-024d-44fa-9fd8-9e2721a5f26e" (UID: "df37e50e-024d-44fa-9fd8-9e2721a5f26e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:39:32.222668 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.222611    2555 scope.go:117] "RemoveContainer" containerID="5569fc54e558a2b8ff5c1b7f09d8c8b4fa10ecb6e89a43bec1b76ae213d38dad"
Jul 28 02:39:32.229171 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.229155    2555 scope.go:117] "RemoveContainer" containerID="6aecc680e94c2bda6eb8a763e6c21bbbba9afad60351cc210ae36bd5ac393757"
Jul 28 02:39:32.229408 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:39:32.229390    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aecc680e94c2bda6eb8a763e6c21bbbba9afad60351cc210ae36bd5ac393757\": container with ID starting with 6aecc680e94c2bda6eb8a763e6c21bbbba9afad60351cc210ae36bd5ac393757 not found: ID does not exist" containerID="6aecc680e94c2bda6eb8a763e6c21bbbba9afad60351cc210ae36bd5ac393757"
Jul 28 02:39:32.229459 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.229416    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aecc680e94c2bda6eb8a763e6c21bbbba9afad60351cc210ae36bd5ac393757"} err="failed to get container status \"6aecc680e94c2bda6eb8a763e6c21bbbba9afad60351cc210ae36bd5ac393757\": rpc error: code = NotFound desc = could not find container \"6aecc680e94c2bda6eb8a763e6c21bbbba9afad60351cc210ae36bd5ac393757\": container with ID starting with 6aecc680e94c2bda6eb8a763e6c21bbbba9afad60351cc210ae36bd5ac393757 not found: ID does not exist"
Jul 28 02:39:32.229459 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.229433    2555 scope.go:117] "RemoveContainer" containerID="e57e46c13b123a18f6d357b510992801cce28fd385f7f657e5d3d10fceecf830"
Jul 28 02:39:32.229668 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:39:32.229651    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57e46c13b123a18f6d357b510992801cce28fd385f7f657e5d3d10fceecf830\": container with ID starting with e57e46c13b123a18f6d357b510992801cce28fd385f7f657e5d3d10fceecf830 not found: ID does not exist" containerID="e57e46c13b123a18f6d357b510992801cce28fd385f7f657e5d3d10fceecf830"
Jul 28 02:39:32.229724 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.229672    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57e46c13b123a18f6d357b510992801cce28fd385f7f657e5d3d10fceecf830"} err="failed to get container status \"e57e46c13b123a18f6d357b510992801cce28fd385f7f657e5d3d10fceecf830\": rpc error: code = NotFound desc = could not find container \"e57e46c13b123a18f6d357b510992801cce28fd385f7f657e5d3d10fceecf830\": container with ID starting with e57e46c13b123a18f6d357b510992801cce28fd385f7f657e5d3d10fceecf830 not found: ID does not exist"
Jul 28 02:39:32.229724 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.229685    2555 scope.go:117] "RemoveContainer" containerID="5569fc54e558a2b8ff5c1b7f09d8c8b4fa10ecb6e89a43bec1b76ae213d38dad"
Jul 28 02:39:32.229892 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:39:32.229874    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5569fc54e558a2b8ff5c1b7f09d8c8b4fa10ecb6e89a43bec1b76ae213d38dad\": container with ID starting with 5569fc54e558a2b8ff5c1b7f09d8c8b4fa10ecb6e89a43bec1b76ae213d38dad not found: ID does not exist" containerID="5569fc54e558a2b8ff5c1b7f09d8c8b4fa10ecb6e89a43bec1b76ae213d38dad"
Jul 28 02:39:32.229946 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.229900    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5569fc54e558a2b8ff5c1b7f09d8c8b4fa10ecb6e89a43bec1b76ae213d38dad"} err="failed to get container status \"5569fc54e558a2b8ff5c1b7f09d8c8b4fa10ecb6e89a43bec1b76ae213d38dad\": rpc error: code = NotFound desc = could not find container \"5569fc54e558a2b8ff5c1b7f09d8c8b4fa10ecb6e89a43bec1b76ae213d38dad\": container with ID starting with 5569fc54e558a2b8ff5c1b7f09d8c8b4fa10ecb6e89a43bec1b76ae213d38dad not found: ID does not exist"
Jul 28 02:39:32.309730 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.309695    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df37e50e-024d-44fa-9fd8-9e2721a5f26e-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:39:32.309730 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.309726    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:39:32.309919 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.309739    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df37e50e-024d-44fa-9fd8-9e2721a5f26e-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:39:32.309919 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.309752    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbvd7\" (UniqueName: \"kubernetes.io/projected/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kube-api-access-gbvd7\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:39:32.309919 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.309763    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df37e50e-024d-44fa-9fd8-9e2721a5f26e-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:39:32.527180 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.527153    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"]
Jul 28 02:39:32.530484 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:32.530462    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-zczfw"]
Jul 28 02:39:33.451534 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:33.451498    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" path="/var/lib/kubelet/pods/df37e50e-024d-44fa-9fd8-9e2721a5f26e/volumes"
Jul 28 02:39:47.213588 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:47.213555    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"]
Jul 28 02:39:47.214050 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:47.213799    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kserve-container" containerID="cri-o://20375b0b2b03aa7850f193a3999438260b2e08f25f5099cbe91f9cb2a2887117" gracePeriod=30
Jul 28 02:39:47.214050 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:47.213835    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kube-rbac-proxy" containerID="cri-o://87d095211eece0070f638a9ca72a9e3a4fc7f3646d8c735578a69722ea0e62e5" gracePeriod=30
Jul 28 02:39:48.260747 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:48.260710    2555 generic.go:356] "Generic (PLEG): container finished" podID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerID="87d095211eece0070f638a9ca72a9e3a4fc7f3646d8c735578a69722ea0e62e5" exitCode=2
Jul 28 02:39:48.261122 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:48.260761    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" event={"ID":"e3f4286d-cae0-4999-aefb-0940f8da5770","Type":"ContainerDied","Data":"87d095211eece0070f638a9ca72a9e3a4fc7f3646d8c735578a69722ea0e62e5"}
Jul 28 02:39:50.141453 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:50.141405    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Jul 28 02:39:55.140974 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:39:55.140928    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Jul 28 02:40:00.141171 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:00.141133    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Jul 28 02:40:00.141586 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:00.141245    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:40:05.141561 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:05.141492    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Jul 28 02:40:10.141752 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:10.141712    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Jul 28 02:40:15.140783 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:15.140741    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Jul 28 02:40:17.360621 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:17.360583    2555 generic.go:356] "Generic (PLEG): container finished" podID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerID="20375b0b2b03aa7850f193a3999438260b2e08f25f5099cbe91f9cb2a2887117" exitCode=137
Jul 28 02:40:17.360988 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:17.360654    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" event={"ID":"e3f4286d-cae0-4999-aefb-0940f8da5770","Type":"ContainerDied","Data":"20375b0b2b03aa7850f193a3999438260b2e08f25f5099cbe91f9cb2a2887117"}
Jul 28 02:40:17.862341 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:17.862318    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:40:18.025644 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.025615    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3f4286d-cae0-4999-aefb-0940f8da5770-cabundle-cert\") pod \"e3f4286d-cae0-4999-aefb-0940f8da5770\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") "
Jul 28 02:40:18.025814 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.025670    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3f4286d-cae0-4999-aefb-0940f8da5770-kserve-provision-location\") pod \"e3f4286d-cae0-4999-aefb-0940f8da5770\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") "
Jul 28 02:40:18.025814 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.025715    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhpqv\" (UniqueName: \"kubernetes.io/projected/e3f4286d-cae0-4999-aefb-0940f8da5770-kube-api-access-dhpqv\") pod \"e3f4286d-cae0-4999-aefb-0940f8da5770\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") "
Jul 28 02:40:18.025814 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.025748    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3f4286d-cae0-4999-aefb-0940f8da5770-proxy-tls\") pod \"e3f4286d-cae0-4999-aefb-0940f8da5770\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") "
Jul 28 02:40:18.025814 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.025806    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3f4286d-cae0-4999-aefb-0940f8da5770-kube-rbac-proxy-sar-config\") pod \"e3f4286d-cae0-4999-aefb-0940f8da5770\" (UID: \"e3f4286d-cae0-4999-aefb-0940f8da5770\") "
Jul 28 02:40:18.026065 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.026040    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f4286d-cae0-4999-aefb-0940f8da5770-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e3f4286d-cae0-4999-aefb-0940f8da5770" (UID: "e3f4286d-cae0-4999-aefb-0940f8da5770"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:40:18.026228 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.026203    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f4286d-cae0-4999-aefb-0940f8da5770-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e3f4286d-cae0-4999-aefb-0940f8da5770" (UID: "e3f4286d-cae0-4999-aefb-0940f8da5770"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:40:18.028035 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.027999    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f4286d-cae0-4999-aefb-0940f8da5770-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e3f4286d-cae0-4999-aefb-0940f8da5770" (UID: "e3f4286d-cae0-4999-aefb-0940f8da5770"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:40:18.028142 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.028092    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f4286d-cae0-4999-aefb-0940f8da5770-kube-api-access-dhpqv" (OuterVolumeSpecName: "kube-api-access-dhpqv") pod "e3f4286d-cae0-4999-aefb-0940f8da5770" (UID: "e3f4286d-cae0-4999-aefb-0940f8da5770"). InnerVolumeSpecName "kube-api-access-dhpqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:40:18.032746 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.032720    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f4286d-cae0-4999-aefb-0940f8da5770-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3f4286d-cae0-4999-aefb-0940f8da5770" (UID: "e3f4286d-cae0-4999-aefb-0940f8da5770"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:40:18.126203 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.126177    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3f4286d-cae0-4999-aefb-0940f8da5770-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:40:18.126203 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.126202    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3f4286d-cae0-4999-aefb-0940f8da5770-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:40:18.126348 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.126214    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3f4286d-cae0-4999-aefb-0940f8da5770-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:40:18.126348 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.126222    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhpqv\" (UniqueName: \"kubernetes.io/projected/e3f4286d-cae0-4999-aefb-0940f8da5770-kube-api-access-dhpqv\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:40:18.126348 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.126232    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3f4286d-cae0-4999-aefb-0940f8da5770-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:40:18.365711 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.365617    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr" event={"ID":"e3f4286d-cae0-4999-aefb-0940f8da5770","Type":"ContainerDied","Data":"849297625bd076c52f752099169df064664bf8fb7f6cef8e8bcec8341f08c12e"}
Jul 28 02:40:18.365711 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.365675    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"
Jul 28 02:40:18.365711 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.365700    2555 scope.go:117] "RemoveContainer" containerID="87d095211eece0070f638a9ca72a9e3a4fc7f3646d8c735578a69722ea0e62e5"
Jul 28 02:40:18.379289 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.379269    2555 scope.go:117] "RemoveContainer" containerID="20375b0b2b03aa7850f193a3999438260b2e08f25f5099cbe91f9cb2a2887117"
Jul 28 02:40:18.386267 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.386245    2555 scope.go:117] "RemoveContainer" containerID="b366310b23e7adbc2347b4cb274075dcc78394cbd0e08fdc8202d21fc4ea021b"
Jul 28 02:40:18.390916 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.390894    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"]
Jul 28 02:40:18.394899 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:18.394881    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-76jzr"]
Jul 28 02:40:19.451261 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:40:19.451225    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" path="/var/lib/kubelet/pods/e3f4286d-cae0-4999-aefb-0940f8da5770/volumes"
Jul 28 02:42:09.930413 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930376    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"]
Jul 28 02:42:09.930873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930674    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kube-rbac-proxy"
Jul 28 02:42:09.930873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930687    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kube-rbac-proxy"
Jul 28 02:42:09.930873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930700    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kserve-container"
Jul 28 02:42:09.930873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930706    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kserve-container"
Jul 28 02:42:09.930873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930715    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="storage-initializer"
Jul 28 02:42:09.930873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930721    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="storage-initializer"
Jul 28 02:42:09.930873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930730    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="storage-initializer"
Jul 28 02:42:09.930873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930735    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="storage-initializer"
Jul 28 02:42:09.930873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930741    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kube-rbac-proxy"
Jul 28 02:42:09.930873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930746    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kube-rbac-proxy"
Jul 28 02:42:09.930873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930755    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kserve-container"
Jul 28 02:42:09.930873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930760    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kserve-container"
Jul 28 02:42:09.930873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930798    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kube-rbac-proxy"
Jul 28 02:42:09.930873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930805    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kube-rbac-proxy"
Jul 28 02:42:09.930873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930810    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f4286d-cae0-4999-aefb-0940f8da5770" containerName="kserve-container"
Jul 28 02:42:09.930873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.930817    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="df37e50e-024d-44fa-9fd8-9e2721a5f26e" containerName="kserve-container"
Jul 28 02:42:09.933734 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.933715    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:09.937465 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.937420    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:42:09.937465 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.937434    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:42:09.937465 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.937420    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:42:09.937753 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.937517    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jul 28 02:42:09.937753 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.937434    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jul 28 02:42:09.937753 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.937421    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:42:09.937753 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.937590    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:42:09.943993 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.943969    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"]
Jul 28 02:42:09.994187 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.994144    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a439aaf-a80f-4e41-bc86-848f941039d5-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-htrjq\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:09.994187 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.994191    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nfft\" (UniqueName: \"kubernetes.io/projected/6a439aaf-a80f-4e41-bc86-848f941039d5-kube-api-access-6nfft\") pod \"isvc-xgboost-predictor-7566d65bf7-htrjq\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:09.994394 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.994228    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a439aaf-a80f-4e41-bc86-848f941039d5-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-htrjq\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:09.994394 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.994270    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a439aaf-a80f-4e41-bc86-848f941039d5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-htrjq\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:09.994394 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:09.994309    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a439aaf-a80f-4e41-bc86-848f941039d5-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-htrjq\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:10.095650 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:10.095610    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a439aaf-a80f-4e41-bc86-848f941039d5-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-htrjq\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:10.095650 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:10.095651    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a439aaf-a80f-4e41-bc86-848f941039d5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-htrjq\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:10.095850 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:10.095677    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a439aaf-a80f-4e41-bc86-848f941039d5-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-htrjq\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:10.095850 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:10.095711    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a439aaf-a80f-4e41-bc86-848f941039d5-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-htrjq\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:10.095850 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:10.095737    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nfft\" (UniqueName: \"kubernetes.io/projected/6a439aaf-a80f-4e41-bc86-848f941039d5-kube-api-access-6nfft\") pod \"isvc-xgboost-predictor-7566d65bf7-htrjq\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:10.096149 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:10.096126    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a439aaf-a80f-4e41-bc86-848f941039d5-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-htrjq\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:10.096363 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:10.096344    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a439aaf-a80f-4e41-bc86-848f941039d5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-htrjq\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:10.096403 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:10.096387    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a439aaf-a80f-4e41-bc86-848f941039d5-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-htrjq\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:10.098334 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:10.098309    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a439aaf-a80f-4e41-bc86-848f941039d5-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-htrjq\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:10.104268 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:10.104244    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nfft\" (UniqueName: \"kubernetes.io/projected/6a439aaf-a80f-4e41-bc86-848f941039d5-kube-api-access-6nfft\") pod \"isvc-xgboost-predictor-7566d65bf7-htrjq\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:10.245035 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:10.244993    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:10.376577 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:10.376552    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"]
Jul 28 02:42:10.378892 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:42:10.378862    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a439aaf_a80f_4e41_bc86_848f941039d5.slice/crio-a64cce0306b10155c2bdc2824f15da1d10f2bf8730f9e87109d1d0a74462b437 WatchSource:0}: Error finding container a64cce0306b10155c2bdc2824f15da1d10f2bf8730f9e87109d1d0a74462b437: Status 404 returned error can't find the container with id a64cce0306b10155c2bdc2824f15da1d10f2bf8730f9e87109d1d0a74462b437
Jul 28 02:42:10.739836 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:10.739801    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" event={"ID":"6a439aaf-a80f-4e41-bc86-848f941039d5","Type":"ContainerStarted","Data":"566e41a7845c458b1bba5c416c2619ec27c8ec309cde2f2c7a0269b1b926d01d"}
Jul 28 02:42:10.740005 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:10.739876    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" event={"ID":"6a439aaf-a80f-4e41-bc86-848f941039d5","Type":"ContainerStarted","Data":"a64cce0306b10155c2bdc2824f15da1d10f2bf8730f9e87109d1d0a74462b437"}
Jul 28 02:42:14.753158 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:14.753120    2555 generic.go:356] "Generic (PLEG): container finished" podID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerID="566e41a7845c458b1bba5c416c2619ec27c8ec309cde2f2c7a0269b1b926d01d" exitCode=0
Jul 28 02:42:14.753598 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:14.753198    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" event={"ID":"6a439aaf-a80f-4e41-bc86-848f941039d5","Type":"ContainerDied","Data":"566e41a7845c458b1bba5c416c2619ec27c8ec309cde2f2c7a0269b1b926d01d"}
Jul 28 02:42:33.829207 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:33.829168    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" event={"ID":"6a439aaf-a80f-4e41-bc86-848f941039d5","Type":"ContainerStarted","Data":"c15cb95302d63356d664c5d8bdb4bbb50dfb630eaffdd5ea8695e6e7065bb89f"}
Jul 28 02:42:33.829651 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:33.829239    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" event={"ID":"6a439aaf-a80f-4e41-bc86-848f941039d5","Type":"ContainerStarted","Data":"447c658d3a710ca31f4870132c18b0ce8a72b0cd5c42a4f1311bf79b0c90e19a"}
Jul 28 02:42:33.829651 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:33.829552    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:33.829759 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:33.829669    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:33.831043 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:33.831016    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 02:42:33.850333 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:33.850276    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" podStartSLOduration=2.744685821 podStartE2EDuration="24.85025618s" totalImagesPullingTime="18.396497763s" totalInitContainerRuntime="3.709072596s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:42:33.849489804 +0000 UTC m=+4600.859008249" watchObservedRunningTime="2026-07-28 02:42:33.85025618 +0000 UTC m=+4600.859774605"
Jul 28 02:42:34.833180 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:34.833137    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 02:42:39.837572 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:39.837543    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:42:39.838230 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:39.838198    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 02:42:49.838719 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:49.838682    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 02:42:59.838392 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:42:59.838349    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 02:43:09.838642 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:09.838596    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 02:43:19.838604 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:19.838554    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 02:43:29.838760 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:29.838720    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 02:43:39.839100 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:39.839061    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:43:50.170193 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:50.170110    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"]
Jul 28 02:43:50.170670 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:50.170369    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="kserve-container" containerID="cri-o://447c658d3a710ca31f4870132c18b0ce8a72b0cd5c42a4f1311bf79b0c90e19a" gracePeriod=30
Jul 28 02:43:50.170670 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:50.170421    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="kube-rbac-proxy" containerID="cri-o://c15cb95302d63356d664c5d8bdb4bbb50dfb630eaffdd5ea8695e6e7065bb89f" gracePeriod=30
Jul 28 02:43:51.091855 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:51.091817    2555 generic.go:356] "Generic (PLEG): container finished" podID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerID="c15cb95302d63356d664c5d8bdb4bbb50dfb630eaffdd5ea8695e6e7065bb89f" exitCode=2
Jul 28 02:43:51.091855 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:51.091859    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" event={"ID":"6a439aaf-a80f-4e41-bc86-848f941039d5","Type":"ContainerDied","Data":"c15cb95302d63356d664c5d8bdb4bbb50dfb630eaffdd5ea8695e6e7065bb89f"}
Jul 28 02:43:53.501569 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:53.501544    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:43:53.649747 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:53.649661    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nfft\" (UniqueName: \"kubernetes.io/projected/6a439aaf-a80f-4e41-bc86-848f941039d5-kube-api-access-6nfft\") pod \"6a439aaf-a80f-4e41-bc86-848f941039d5\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") "
Jul 28 02:43:53.649747 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:53.649698    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a439aaf-a80f-4e41-bc86-848f941039d5-cabundle-cert\") pod \"6a439aaf-a80f-4e41-bc86-848f941039d5\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") "
Jul 28 02:43:53.649747 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:53.649729    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a439aaf-a80f-4e41-bc86-848f941039d5-proxy-tls\") pod \"6a439aaf-a80f-4e41-bc86-848f941039d5\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") "
Jul 28 02:43:53.650039 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:53.649759    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a439aaf-a80f-4e41-bc86-848f941039d5-kube-rbac-proxy-sar-config\") pod \"6a439aaf-a80f-4e41-bc86-848f941039d5\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") "
Jul 28 02:43:53.650039 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:53.649798    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a439aaf-a80f-4e41-bc86-848f941039d5-kserve-provision-location\") pod \"6a439aaf-a80f-4e41-bc86-848f941039d5\" (UID: \"6a439aaf-a80f-4e41-bc86-848f941039d5\") "
Jul 28 02:43:53.650115 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:53.650063    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a439aaf-a80f-4e41-bc86-848f941039d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a439aaf-a80f-4e41-bc86-848f941039d5" (UID: "6a439aaf-a80f-4e41-bc86-848f941039d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:43:53.650162 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:53.650106    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a439aaf-a80f-4e41-bc86-848f941039d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a439aaf-a80f-4e41-bc86-848f941039d5" (UID: "6a439aaf-a80f-4e41-bc86-848f941039d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:43:53.650162 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:53.650122    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a439aaf-a80f-4e41-bc86-848f941039d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a439aaf-a80f-4e41-bc86-848f941039d5" (UID: "6a439aaf-a80f-4e41-bc86-848f941039d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:43:53.651971 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:53.651940    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a439aaf-a80f-4e41-bc86-848f941039d5-kube-api-access-6nfft" (OuterVolumeSpecName: "kube-api-access-6nfft") pod "6a439aaf-a80f-4e41-bc86-848f941039d5" (UID: "6a439aaf-a80f-4e41-bc86-848f941039d5"). InnerVolumeSpecName "kube-api-access-6nfft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:43:53.652126 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:53.651973    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a439aaf-a80f-4e41-bc86-848f941039d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a439aaf-a80f-4e41-bc86-848f941039d5" (UID: "6a439aaf-a80f-4e41-bc86-848f941039d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:43:53.750333 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:53.750308    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a439aaf-a80f-4e41-bc86-848f941039d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:43:53.750333 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:53.750331    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a439aaf-a80f-4e41-bc86-848f941039d5-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:43:53.750500 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:53.750340    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nfft\" (UniqueName: \"kubernetes.io/projected/6a439aaf-a80f-4e41-bc86-848f941039d5-kube-api-access-6nfft\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:43:53.750500 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:53.750348    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a439aaf-a80f-4e41-bc86-848f941039d5-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:43:53.750500 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:53.750357    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a439aaf-a80f-4e41-bc86-848f941039d5-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:43:54.103612 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:54.103582    2555 generic.go:356] "Generic (PLEG): container finished" podID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerID="447c658d3a710ca31f4870132c18b0ce8a72b0cd5c42a4f1311bf79b0c90e19a" exitCode=0
Jul 28 02:43:54.103792 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:54.103646    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" event={"ID":"6a439aaf-a80f-4e41-bc86-848f941039d5","Type":"ContainerDied","Data":"447c658d3a710ca31f4870132c18b0ce8a72b0cd5c42a4f1311bf79b0c90e19a"}
Jul 28 02:43:54.103792 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:54.103656    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"
Jul 28 02:43:54.103792 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:54.103718    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq" event={"ID":"6a439aaf-a80f-4e41-bc86-848f941039d5","Type":"ContainerDied","Data":"a64cce0306b10155c2bdc2824f15da1d10f2bf8730f9e87109d1d0a74462b437"}
Jul 28 02:43:54.103792 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:54.103747    2555 scope.go:117] "RemoveContainer" containerID="c15cb95302d63356d664c5d8bdb4bbb50dfb630eaffdd5ea8695e6e7065bb89f"
Jul 28 02:43:54.111833 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:54.111815    2555 scope.go:117] "RemoveContainer" containerID="447c658d3a710ca31f4870132c18b0ce8a72b0cd5c42a4f1311bf79b0c90e19a"
Jul 28 02:43:54.118940 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:54.118922    2555 scope.go:117] "RemoveContainer" containerID="566e41a7845c458b1bba5c416c2619ec27c8ec309cde2f2c7a0269b1b926d01d"
Jul 28 02:43:54.126015 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:54.125989    2555 scope.go:117] "RemoveContainer" containerID="c15cb95302d63356d664c5d8bdb4bbb50dfb630eaffdd5ea8695e6e7065bb89f"
Jul 28 02:43:54.126265 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:43:54.126248    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15cb95302d63356d664c5d8bdb4bbb50dfb630eaffdd5ea8695e6e7065bb89f\": container with ID starting with c15cb95302d63356d664c5d8bdb4bbb50dfb630eaffdd5ea8695e6e7065bb89f not found: ID does not exist" containerID="c15cb95302d63356d664c5d8bdb4bbb50dfb630eaffdd5ea8695e6e7065bb89f"
Jul 28 02:43:54.126316 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:54.126274    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15cb95302d63356d664c5d8bdb4bbb50dfb630eaffdd5ea8695e6e7065bb89f"} err="failed to get container status \"c15cb95302d63356d664c5d8bdb4bbb50dfb630eaffdd5ea8695e6e7065bb89f\": rpc error: code = NotFound desc = could not find container \"c15cb95302d63356d664c5d8bdb4bbb50dfb630eaffdd5ea8695e6e7065bb89f\": container with ID starting with c15cb95302d63356d664c5d8bdb4bbb50dfb630eaffdd5ea8695e6e7065bb89f not found: ID does not exist"
Jul 28 02:43:54.126316 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:54.126290    2555 scope.go:117] "RemoveContainer" containerID="447c658d3a710ca31f4870132c18b0ce8a72b0cd5c42a4f1311bf79b0c90e19a"
Jul 28 02:43:54.126476 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:43:54.126461    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447c658d3a710ca31f4870132c18b0ce8a72b0cd5c42a4f1311bf79b0c90e19a\": container with ID starting with 447c658d3a710ca31f4870132c18b0ce8a72b0cd5c42a4f1311bf79b0c90e19a not found: ID does not exist" containerID="447c658d3a710ca31f4870132c18b0ce8a72b0cd5c42a4f1311bf79b0c90e19a"
Jul 28 02:43:54.126513 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:54.126479    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447c658d3a710ca31f4870132c18b0ce8a72b0cd5c42a4f1311bf79b0c90e19a"} err="failed to get container status \"447c658d3a710ca31f4870132c18b0ce8a72b0cd5c42a4f1311bf79b0c90e19a\": rpc error: code = NotFound desc = could not find container \"447c658d3a710ca31f4870132c18b0ce8a72b0cd5c42a4f1311bf79b0c90e19a\": container with ID starting with 447c658d3a710ca31f4870132c18b0ce8a72b0cd5c42a4f1311bf79b0c90e19a not found: ID does not exist"
Jul 28 02:43:54.126513 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:54.126491    2555 scope.go:117] "RemoveContainer" containerID="566e41a7845c458b1bba5c416c2619ec27c8ec309cde2f2c7a0269b1b926d01d"
Jul 28 02:43:54.126740 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:43:54.126720    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566e41a7845c458b1bba5c416c2619ec27c8ec309cde2f2c7a0269b1b926d01d\": container with ID starting with 566e41a7845c458b1bba5c416c2619ec27c8ec309cde2f2c7a0269b1b926d01d not found: ID does not exist" containerID="566e41a7845c458b1bba5c416c2619ec27c8ec309cde2f2c7a0269b1b926d01d"
Jul 28 02:43:54.126834 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:54.126744    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566e41a7845c458b1bba5c416c2619ec27c8ec309cde2f2c7a0269b1b926d01d"} err="failed to get container status \"566e41a7845c458b1bba5c416c2619ec27c8ec309cde2f2c7a0269b1b926d01d\": rpc error: code = NotFound desc = could not find container \"566e41a7845c458b1bba5c416c2619ec27c8ec309cde2f2c7a0269b1b926d01d\": container with ID starting with 566e41a7845c458b1bba5c416c2619ec27c8ec309cde2f2c7a0269b1b926d01d not found: ID does not exist"
Jul 28 02:43:54.129067 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:54.129047    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"]
Jul 28 02:43:54.146302 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:54.146280    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-htrjq"]
Jul 28 02:43:55.397313 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.397278    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"]
Jul 28 02:43:55.397704 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.397511    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="kserve-container"
Jul 28 02:43:55.397704 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.397540    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="kserve-container"
Jul 28 02:43:55.397704 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.397557    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="kube-rbac-proxy"
Jul 28 02:43:55.397704 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.397565    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="kube-rbac-proxy"
Jul 28 02:43:55.397704 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.397581    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="storage-initializer"
Jul 28 02:43:55.397704 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.397588    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="storage-initializer"
Jul 28 02:43:55.397704 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.397643    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="kube-rbac-proxy"
Jul 28 02:43:55.397704 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.397654    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" containerName="kserve-container"
Jul 28 02:43:55.400849 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.400829    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.403457 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.403436    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jul 28 02:43:55.407257 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.407232    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 02:43:55.407257 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.407251    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:43:55.407422 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.407328    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:43:55.407422 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.407251    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:43:55.407422 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.407377    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:43:55.407611 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.407246    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:43:55.430420 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.430401    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"]
Jul 28 02:43:55.452197 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.452167    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a439aaf-a80f-4e41-bc86-848f941039d5" path="/var/lib/kubelet/pods/6a439aaf-a80f-4e41-bc86-848f941039d5/volumes"
Jul 28 02:43:55.463825 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.463805    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.463927 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.463836    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.463927 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.463859    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdnv6\" (UniqueName: \"kubernetes.io/projected/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kube-api-access-mdnv6\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.463927 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.463880    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.464056 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.463948    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.565311 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.565237    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdnv6\" (UniqueName: \"kubernetes.io/projected/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kube-api-access-mdnv6\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.565311 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.565274    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.565513 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.565314    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.565615 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.565516    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.565615 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.565590    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.565927 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.565900    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.566038 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.566020    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.566174 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.566158    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.568019 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.568000    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.575724 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.575697    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdnv6\" (UniqueName: \"kubernetes.io/projected/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kube-api-access-mdnv6\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.710401 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.710370    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:43:55.838985 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:55.838923    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"]
Jul 28 02:43:55.841563 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:43:55.841535    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c8d218_1095_42a0_8d29_1a7c9c58e35d.slice/crio-26b07969c012467cc93099291b148bd95be79d9312eaa971d2bf7195a2518a10 WatchSource:0}: Error finding container 26b07969c012467cc93099291b148bd95be79d9312eaa971d2bf7195a2518a10: Status 404 returned error can't find the container with id 26b07969c012467cc93099291b148bd95be79d9312eaa971d2bf7195a2518a10
Jul 28 02:43:56.113373 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:56.113282    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7" event={"ID":"b0c8d218-1095-42a0-8d29-1a7c9c58e35d","Type":"ContainerStarted","Data":"7802d49c2a4c41c1c8c604c7d9d6ce2d07aadcf7c1c8552cea485277db569686"}
Jul 28 02:43:56.113373 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:43:56.113350    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7" event={"ID":"b0c8d218-1095-42a0-8d29-1a7c9c58e35d","Type":"ContainerStarted","Data":"26b07969c012467cc93099291b148bd95be79d9312eaa971d2bf7195a2518a10"}
Jul 28 02:44:00.129263 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:00.129225    2555 generic.go:356] "Generic (PLEG): container finished" podID="b0c8d218-1095-42a0-8d29-1a7c9c58e35d" containerID="7802d49c2a4c41c1c8c604c7d9d6ce2d07aadcf7c1c8552cea485277db569686" exitCode=0
Jul 28 02:44:00.129655 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:00.129281    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7" event={"ID":"b0c8d218-1095-42a0-8d29-1a7c9c58e35d","Type":"ContainerDied","Data":"7802d49c2a4c41c1c8c604c7d9d6ce2d07aadcf7c1c8552cea485277db569686"}
Jul 28 02:44:01.134821 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:01.134786    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7" event={"ID":"b0c8d218-1095-42a0-8d29-1a7c9c58e35d","Type":"ContainerStarted","Data":"9e3759bb74f743c1f6be0864627d11082dde5752a0fad9edf05b6ca54022396b"}
Jul 28 02:44:01.135270 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:01.134844    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7" event={"ID":"b0c8d218-1095-42a0-8d29-1a7c9c58e35d","Type":"ContainerStarted","Data":"1c06f90eec299d3c2db0af88d0fe660d1faa536842cf2bbce33e9ec78fa07221"}
Jul 28 02:44:01.135270 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:01.135109    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:44:01.135270 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:01.135176    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:44:01.164394 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:01.164346    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7" podStartSLOduration=2.599618838 podStartE2EDuration="6.164331403s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.564712565s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:44:01.161277276 +0000 UTC m=+4688.170795712" watchObservedRunningTime="2026-07-28 02:44:01.164331403 +0000 UTC m=+4688.173849828"
Jul 28 02:44:07.143700 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:07.143669    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:44:37.147317 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:37.147286    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:44:45.342056 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:45.342018    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"]
Jul 28 02:44:45.342442 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:45.342247    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7" podUID="b0c8d218-1095-42a0-8d29-1a7c9c58e35d" containerName="kserve-container" containerID="cri-o://1c06f90eec299d3c2db0af88d0fe660d1faa536842cf2bbce33e9ec78fa07221" gracePeriod=30
Jul 28 02:44:45.342442 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:45.342288    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7" podUID="b0c8d218-1095-42a0-8d29-1a7c9c58e35d" containerName="kube-rbac-proxy" containerID="cri-o://9e3759bb74f743c1f6be0864627d11082dde5752a0fad9edf05b6ca54022396b" gracePeriod=30
Jul 28 02:44:46.285977 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:46.285944    2555 generic.go:356] "Generic (PLEG): container finished" podID="b0c8d218-1095-42a0-8d29-1a7c9c58e35d" containerID="9e3759bb74f743c1f6be0864627d11082dde5752a0fad9edf05b6ca54022396b" exitCode=2
Jul 28 02:44:46.286148 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:46.286015    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7" event={"ID":"b0c8d218-1095-42a0-8d29-1a7c9c58e35d","Type":"ContainerDied","Data":"9e3759bb74f743c1f6be0864627d11082dde5752a0fad9edf05b6ca54022396b"}
Jul 28 02:44:47.139058 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:47.139014    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7" podUID="b0c8d218-1095-42a0-8d29-1a7c9c58e35d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.63:8643/healthz\": dial tcp 10.133.0.63:8643: connect: connection refused"
Jul 28 02:44:50.533970 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.533925    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"]
Jul 28 02:44:50.537300 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.537283    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:50.540344 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.540310    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 02:44:50.540499 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.540342    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jul 28 02:44:50.540499 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.540386    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0632e025-c6b0-4a6a-92c4-9e3c793ed001-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bszzz\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:50.540499 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.540446    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bszzz\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:50.545454 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.540778    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bszzz\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:50.545454 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.540829    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzzh2\" (UniqueName: \"kubernetes.io/projected/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kube-api-access-zzzh2\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bszzz\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:50.545454 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.540892    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0632e025-c6b0-4a6a-92c4-9e3c793ed001-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bszzz\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:50.551341 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.551317    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"]
Jul 28 02:44:50.641500 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.641471    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bszzz\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:50.641500 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.641504    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzzh2\" (UniqueName: \"kubernetes.io/projected/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kube-api-access-zzzh2\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bszzz\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:50.641757 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.641561    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0632e025-c6b0-4a6a-92c4-9e3c793ed001-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bszzz\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:50.641757 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.641588    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0632e025-c6b0-4a6a-92c4-9e3c793ed001-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bszzz\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:50.641757 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.641613    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bszzz\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:50.641865 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:44:50.641760    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 28 02:44:50.641865 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:44:50.641829    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0632e025-c6b0-4a6a-92c4-9e3c793ed001-proxy-tls podName:0632e025-c6b0-4a6a-92c4-9e3c793ed001 nodeName:}" failed. No retries permitted until 2026-07-28 02:44:51.141808764 +0000 UTC m=+4738.151327170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0632e025-c6b0-4a6a-92c4-9e3c793ed001-proxy-tls") pod "xgboost-v2-mlserver-predictor-59944bcf4c-bszzz" (UID: "0632e025-c6b0-4a6a-92c4-9e3c793ed001") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 28 02:44:50.642009 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.641989    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bszzz\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:50.642144 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.642125    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bszzz\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:50.642225 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.642210    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0632e025-c6b0-4a6a-92c4-9e3c793ed001-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bszzz\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:50.652645 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:50.652619    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzzh2\" (UniqueName: \"kubernetes.io/projected/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kube-api-access-zzzh2\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bszzz\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:51.145334 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.145304    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0632e025-c6b0-4a6a-92c4-9e3c793ed001-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bszzz\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:51.147785 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.147760    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0632e025-c6b0-4a6a-92c4-9e3c793ed001-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bszzz\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:51.149632 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.149615    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:51.289745 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.289715    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"]
Jul 28 02:44:51.292148 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:44:51.292122    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0632e025_c6b0_4a6a_92c4_9e3c793ed001.slice/crio-b5eb20466e1bac207d4d8b8396e6b5f340ba77269b0af6850e01974f9f1ea78e WatchSource:0}: Error finding container b5eb20466e1bac207d4d8b8396e6b5f340ba77269b0af6850e01974f9f1ea78e: Status 404 returned error can't find the container with id b5eb20466e1bac207d4d8b8396e6b5f340ba77269b0af6850e01974f9f1ea78e
Jul 28 02:44:51.305391 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.305271    2555 generic.go:356] "Generic (PLEG): container finished" podID="b0c8d218-1095-42a0-8d29-1a7c9c58e35d" containerID="1c06f90eec299d3c2db0af88d0fe660d1faa536842cf2bbce33e9ec78fa07221" exitCode=0
Jul 28 02:44:51.305391 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.305332    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7" event={"ID":"b0c8d218-1095-42a0-8d29-1a7c9c58e35d","Type":"ContainerDied","Data":"1c06f90eec299d3c2db0af88d0fe660d1faa536842cf2bbce33e9ec78fa07221"}
Jul 28 02:44:51.306605 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.306569    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz" event={"ID":"0632e025-c6b0-4a6a-92c4-9e3c793ed001","Type":"ContainerStarted","Data":"b5eb20466e1bac207d4d8b8396e6b5f340ba77269b0af6850e01974f9f1ea78e"}
Jul 28 02:44:51.373990 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.373968    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:44:51.447780 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.447707    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kserve-provision-location\") pod \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") "
Jul 28 02:44:51.447780 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.447741    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-proxy-tls\") pod \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") "
Jul 28 02:44:51.447780 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.447779    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdnv6\" (UniqueName: \"kubernetes.io/projected/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kube-api-access-mdnv6\") pod \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") "
Jul 28 02:44:51.448037 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.447800    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-cabundle-cert\") pod \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") "
Jul 28 02:44:51.448037 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.447823    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kube-rbac-proxy-sar-config\") pod \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\" (UID: \"b0c8d218-1095-42a0-8d29-1a7c9c58e35d\") "
Jul 28 02:44:51.448184 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.448152    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b0c8d218-1095-42a0-8d29-1a7c9c58e35d" (UID: "b0c8d218-1095-42a0-8d29-1a7c9c58e35d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:44:51.448248 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.448185    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b0c8d218-1095-42a0-8d29-1a7c9c58e35d" (UID: "b0c8d218-1095-42a0-8d29-1a7c9c58e35d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:44:51.448248 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.448219    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b0c8d218-1095-42a0-8d29-1a7c9c58e35d" (UID: "b0c8d218-1095-42a0-8d29-1a7c9c58e35d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:44:51.450210 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.450182    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0c8d218-1095-42a0-8d29-1a7c9c58e35d" (UID: "b0c8d218-1095-42a0-8d29-1a7c9c58e35d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:44:51.450210 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.450204    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kube-api-access-mdnv6" (OuterVolumeSpecName: "kube-api-access-mdnv6") pod "b0c8d218-1095-42a0-8d29-1a7c9c58e35d" (UID: "b0c8d218-1095-42a0-8d29-1a7c9c58e35d"). InnerVolumeSpecName "kube-api-access-mdnv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:44:51.548454 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.548425    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:44:51.548454 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.548453    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:44:51.548844 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.548462    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdnv6\" (UniqueName: \"kubernetes.io/projected/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kube-api-access-mdnv6\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:44:51.548844 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.548472    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:44:51.548844 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:51.548480    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c8d218-1095-42a0-8d29-1a7c9c58e35d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:44:52.311991 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:52.311947    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7" event={"ID":"b0c8d218-1095-42a0-8d29-1a7c9c58e35d","Type":"ContainerDied","Data":"26b07969c012467cc93099291b148bd95be79d9312eaa971d2bf7195a2518a10"}
Jul 28 02:44:52.311991 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:52.311969    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"
Jul 28 02:44:52.312226 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:52.312016    2555 scope.go:117] "RemoveContainer" containerID="9e3759bb74f743c1f6be0864627d11082dde5752a0fad9edf05b6ca54022396b"
Jul 28 02:44:52.313768 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:52.313743    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz" event={"ID":"0632e025-c6b0-4a6a-92c4-9e3c793ed001","Type":"ContainerStarted","Data":"eb36ff4ae5cef55e69fb07c67b6df6c9b54fc82a99db6be1babaf09c17d140c9"}
Jul 28 02:44:52.320182 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:52.320163    2555 scope.go:117] "RemoveContainer" containerID="1c06f90eec299d3c2db0af88d0fe660d1faa536842cf2bbce33e9ec78fa07221"
Jul 28 02:44:52.327064 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:52.327049    2555 scope.go:117] "RemoveContainer" containerID="7802d49c2a4c41c1c8c604c7d9d6ce2d07aadcf7c1c8552cea485277db569686"
Jul 28 02:44:52.332236 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:52.332210    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"]
Jul 28 02:44:52.337311 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:52.337291    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hz5s7"]
Jul 28 02:44:53.451684 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:53.451652    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0c8d218-1095-42a0-8d29-1a7c9c58e35d" path="/var/lib/kubelet/pods/b0c8d218-1095-42a0-8d29-1a7c9c58e35d/volumes"
Jul 28 02:44:55.325058 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:55.325031    2555 generic.go:356] "Generic (PLEG): container finished" podID="0632e025-c6b0-4a6a-92c4-9e3c793ed001" containerID="eb36ff4ae5cef55e69fb07c67b6df6c9b54fc82a99db6be1babaf09c17d140c9" exitCode=0
Jul 28 02:44:55.325437 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:55.325087    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz" event={"ID":"0632e025-c6b0-4a6a-92c4-9e3c793ed001","Type":"ContainerDied","Data":"eb36ff4ae5cef55e69fb07c67b6df6c9b54fc82a99db6be1babaf09c17d140c9"}
Jul 28 02:44:56.330484 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:56.330447    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz" event={"ID":"0632e025-c6b0-4a6a-92c4-9e3c793ed001","Type":"ContainerStarted","Data":"be5a9ae39a6666c4caede1b66a638f112458b912d83f73b8c2253c6f1aad847b"}
Jul 28 02:44:56.330894 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:56.330510    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz" event={"ID":"0632e025-c6b0-4a6a-92c4-9e3c793ed001","Type":"ContainerStarted","Data":"2860f74c2c4663f131d177a1e408db672d11ee86648ae3587702480ecf74fc6c"}
Jul 28 02:44:56.330894 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:56.330830    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:56.330894 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:56.330882    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:44:56.368419 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:44:56.368362    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz" podStartSLOduration=2.506638596 podStartE2EDuration="6.368345447s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.861706851s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:44:56.362236081 +0000 UTC m=+4743.371754506" watchObservedRunningTime="2026-07-28 02:44:56.368345447 +0000 UTC m=+4743.377863871"
Jul 28 02:45:02.339510 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:02.339478    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:45:32.343765 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:32.343692    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:45:40.580184 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:40.580151    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"]
Jul 28 02:45:40.580749 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:40.580401    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz" podUID="0632e025-c6b0-4a6a-92c4-9e3c793ed001" containerName="kserve-container" containerID="cri-o://2860f74c2c4663f131d177a1e408db672d11ee86648ae3587702480ecf74fc6c" gracePeriod=30
Jul 28 02:45:40.580749 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:40.580435    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz" podUID="0632e025-c6b0-4a6a-92c4-9e3c793ed001" containerName="kube-rbac-proxy" containerID="cri-o://be5a9ae39a6666c4caede1b66a638f112458b912d83f73b8c2253c6f1aad847b" gracePeriod=30
Jul 28 02:45:41.492059 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:41.492019    2555 generic.go:356] "Generic (PLEG): container finished" podID="0632e025-c6b0-4a6a-92c4-9e3c793ed001" containerID="be5a9ae39a6666c4caede1b66a638f112458b912d83f73b8c2253c6f1aad847b" exitCode=2
Jul 28 02:45:41.492059 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:41.492058    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz" event={"ID":"0632e025-c6b0-4a6a-92c4-9e3c793ed001","Type":"ContainerDied","Data":"be5a9ae39a6666c4caede1b66a638f112458b912d83f73b8c2253c6f1aad847b"}
Jul 28 02:45:42.334948 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:42.334899    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz" podUID="0632e025-c6b0-4a6a-92c4-9e3c793ed001" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.64:8643/healthz\": dial tcp 10.133.0.64:8643: connect: connection refused"
Jul 28 02:45:45.759600 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.759559    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"]
Jul 28 02:45:45.759987 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.759823    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c8d218-1095-42a0-8d29-1a7c9c58e35d" containerName="kube-rbac-proxy"
Jul 28 02:45:45.759987 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.759834    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c8d218-1095-42a0-8d29-1a7c9c58e35d" containerName="kube-rbac-proxy"
Jul 28 02:45:45.759987 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.759843    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c8d218-1095-42a0-8d29-1a7c9c58e35d" containerName="kserve-container"
Jul 28 02:45:45.759987 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.759849    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c8d218-1095-42a0-8d29-1a7c9c58e35d" containerName="kserve-container"
Jul 28 02:45:45.759987 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.759857    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c8d218-1095-42a0-8d29-1a7c9c58e35d" containerName="storage-initializer"
Jul 28 02:45:45.759987 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.759863    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c8d218-1095-42a0-8d29-1a7c9c58e35d" containerName="storage-initializer"
Jul 28 02:45:45.759987 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.759932    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0c8d218-1095-42a0-8d29-1a7c9c58e35d" containerName="kube-rbac-proxy"
Jul 28 02:45:45.759987 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.759941    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0c8d218-1095-42a0-8d29-1a7c9c58e35d" containerName="kserve-container"
Jul 28 02:45:45.763017 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.763000    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:45.765624 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.765599    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jul 28 02:45:45.765795 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.765688    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jul 28 02:45:45.777271 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.777240    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"]
Jul 28 02:45:45.850736 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.850698    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4782d67b-d390-49de-a187-6860617ea785-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v87jh\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:45.850736 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.850739    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4782d67b-d390-49de-a187-6860617ea785-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v87jh\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:45.850982 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.850761    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4782d67b-d390-49de-a187-6860617ea785-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v87jh\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:45.850982 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.850842    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4782d67b-d390-49de-a187-6860617ea785-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v87jh\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:45.850982 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.850894    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hrtr\" (UniqueName: \"kubernetes.io/projected/4782d67b-d390-49de-a187-6860617ea785-kube-api-access-9hrtr\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v87jh\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:45.952088 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.952046    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hrtr\" (UniqueName: \"kubernetes.io/projected/4782d67b-d390-49de-a187-6860617ea785-kube-api-access-9hrtr\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v87jh\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:45.952267 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.952099    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4782d67b-d390-49de-a187-6860617ea785-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v87jh\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:45.952267 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.952121    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4782d67b-d390-49de-a187-6860617ea785-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v87jh\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:45.952267 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.952141    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4782d67b-d390-49de-a187-6860617ea785-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v87jh\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:45.952267 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.952165    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4782d67b-d390-49de-a187-6860617ea785-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v87jh\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:45.952449 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:45:45.952419    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Jul 28 02:45:45.952542 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:45:45.952504    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4782d67b-d390-49de-a187-6860617ea785-proxy-tls podName:4782d67b-d390-49de-a187-6860617ea785 nodeName:}" failed. No retries permitted until 2026-07-28 02:45:46.452485738 +0000 UTC m=+4793.462004142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4782d67b-d390-49de-a187-6860617ea785-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-v87jh" (UID: "4782d67b-d390-49de-a187-6860617ea785") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Jul 28 02:45:45.952611 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.952594    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4782d67b-d390-49de-a187-6860617ea785-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v87jh\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:45.952902 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.952887    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4782d67b-d390-49de-a187-6860617ea785-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v87jh\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:45.952951 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.952913    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4782d67b-d390-49de-a187-6860617ea785-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v87jh\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:45.964454 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:45.964425    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hrtr\" (UniqueName: \"kubernetes.io/projected/4782d67b-d390-49de-a187-6860617ea785-kube-api-access-9hrtr\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v87jh\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:46.455421 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:46.455383    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4782d67b-d390-49de-a187-6860617ea785-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v87jh\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:46.458177 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:46.458151    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4782d67b-d390-49de-a187-6860617ea785-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v87jh\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:46.673031 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:46.672988    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:46.805199 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:46.805113    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"]
Jul 28 02:45:46.807935 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:45:46.807908    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4782d67b_d390_49de_a187_6860617ea785.slice/crio-83c0fbefbe02f4c759ba4ecd574e66ec03277813bfa794a662855f3f97e0d579 WatchSource:0}: Error finding container 83c0fbefbe02f4c759ba4ecd574e66ec03277813bfa794a662855f3f97e0d579: Status 404 returned error can't find the container with id 83c0fbefbe02f4c759ba4ecd574e66ec03277813bfa794a662855f3f97e0d579
Jul 28 02:45:47.219557 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.219516    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:45:47.364886 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.364788    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kserve-provision-location\") pod \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") "
Jul 28 02:45:47.364886 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.364844    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0632e025-c6b0-4a6a-92c4-9e3c793ed001-cabundle-cert\") pod \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") "
Jul 28 02:45:47.364886 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.364880    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzzh2\" (UniqueName: \"kubernetes.io/projected/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kube-api-access-zzzh2\") pod \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") "
Jul 28 02:45:47.365178 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.364901    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kube-rbac-proxy-sar-config\") pod \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") "
Jul 28 02:45:47.365178 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.364934    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0632e025-c6b0-4a6a-92c4-9e3c793ed001-proxy-tls\") pod \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\" (UID: \"0632e025-c6b0-4a6a-92c4-9e3c793ed001\") "
Jul 28 02:45:47.365286 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.365214    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0632e025-c6b0-4a6a-92c4-9e3c793ed001" (UID: "0632e025-c6b0-4a6a-92c4-9e3c793ed001"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:45:47.365333 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.365279    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0632e025-c6b0-4a6a-92c4-9e3c793ed001-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0632e025-c6b0-4a6a-92c4-9e3c793ed001" (UID: "0632e025-c6b0-4a6a-92c4-9e3c793ed001"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:45:47.365333 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.365274    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0632e025-c6b0-4a6a-92c4-9e3c793ed001" (UID: "0632e025-c6b0-4a6a-92c4-9e3c793ed001"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:45:47.367220 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.367181    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kube-api-access-zzzh2" (OuterVolumeSpecName: "kube-api-access-zzzh2") pod "0632e025-c6b0-4a6a-92c4-9e3c793ed001" (UID: "0632e025-c6b0-4a6a-92c4-9e3c793ed001"). InnerVolumeSpecName "kube-api-access-zzzh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:45:47.367331 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.367223    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0632e025-c6b0-4a6a-92c4-9e3c793ed001-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0632e025-c6b0-4a6a-92c4-9e3c793ed001" (UID: "0632e025-c6b0-4a6a-92c4-9e3c793ed001"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:45:47.465567 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.465542    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:45:47.465567 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.465567    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0632e025-c6b0-4a6a-92c4-9e3c793ed001-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:45:47.465717 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.465577    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzzh2\" (UniqueName: \"kubernetes.io/projected/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kube-api-access-zzzh2\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:45:47.465717 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.465586    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0632e025-c6b0-4a6a-92c4-9e3c793ed001-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:45:47.465717 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.465595    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0632e025-c6b0-4a6a-92c4-9e3c793ed001-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:45:47.514249 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.514211    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" event={"ID":"4782d67b-d390-49de-a187-6860617ea785","Type":"ContainerStarted","Data":"79b622450068b3af1406387edc13915e2f26049c6f79362b9baa58b4889681a7"}
Jul 28 02:45:47.514391 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.514271    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" event={"ID":"4782d67b-d390-49de-a187-6860617ea785","Type":"ContainerStarted","Data":"83c0fbefbe02f4c759ba4ecd574e66ec03277813bfa794a662855f3f97e0d579"}
Jul 28 02:45:47.516155 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.516130    2555 generic.go:356] "Generic (PLEG): container finished" podID="0632e025-c6b0-4a6a-92c4-9e3c793ed001" containerID="2860f74c2c4663f131d177a1e408db672d11ee86648ae3587702480ecf74fc6c" exitCode=0
Jul 28 02:45:47.516250 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.516184    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"
Jul 28 02:45:47.516250 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.516206    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz" event={"ID":"0632e025-c6b0-4a6a-92c4-9e3c793ed001","Type":"ContainerDied","Data":"2860f74c2c4663f131d177a1e408db672d11ee86648ae3587702480ecf74fc6c"}
Jul 28 02:45:47.516339 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.516257    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz" event={"ID":"0632e025-c6b0-4a6a-92c4-9e3c793ed001","Type":"ContainerDied","Data":"b5eb20466e1bac207d4d8b8396e6b5f340ba77269b0af6850e01974f9f1ea78e"}
Jul 28 02:45:47.516339 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.516284    2555 scope.go:117] "RemoveContainer" containerID="be5a9ae39a6666c4caede1b66a638f112458b912d83f73b8c2253c6f1aad847b"
Jul 28 02:45:47.523880 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.523862    2555 scope.go:117] "RemoveContainer" containerID="2860f74c2c4663f131d177a1e408db672d11ee86648ae3587702480ecf74fc6c"
Jul 28 02:45:47.530913 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.530896    2555 scope.go:117] "RemoveContainer" containerID="eb36ff4ae5cef55e69fb07c67b6df6c9b54fc82a99db6be1babaf09c17d140c9"
Jul 28 02:45:47.538510 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.538485    2555 scope.go:117] "RemoveContainer" containerID="be5a9ae39a6666c4caede1b66a638f112458b912d83f73b8c2253c6f1aad847b"
Jul 28 02:45:47.538789 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:45:47.538771    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5a9ae39a6666c4caede1b66a638f112458b912d83f73b8c2253c6f1aad847b\": container with ID starting with be5a9ae39a6666c4caede1b66a638f112458b912d83f73b8c2253c6f1aad847b not found: ID does not exist" containerID="be5a9ae39a6666c4caede1b66a638f112458b912d83f73b8c2253c6f1aad847b"
Jul 28 02:45:47.538842 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.538796    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5a9ae39a6666c4caede1b66a638f112458b912d83f73b8c2253c6f1aad847b"} err="failed to get container status \"be5a9ae39a6666c4caede1b66a638f112458b912d83f73b8c2253c6f1aad847b\": rpc error: code = NotFound desc = could not find container \"be5a9ae39a6666c4caede1b66a638f112458b912d83f73b8c2253c6f1aad847b\": container with ID starting with be5a9ae39a6666c4caede1b66a638f112458b912d83f73b8c2253c6f1aad847b not found: ID does not exist"
Jul 28 02:45:47.538842 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.538812    2555 scope.go:117] "RemoveContainer" containerID="2860f74c2c4663f131d177a1e408db672d11ee86648ae3587702480ecf74fc6c"
Jul 28 02:45:47.539073 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:45:47.539051    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2860f74c2c4663f131d177a1e408db672d11ee86648ae3587702480ecf74fc6c\": container with ID starting with 2860f74c2c4663f131d177a1e408db672d11ee86648ae3587702480ecf74fc6c not found: ID does not exist" containerID="2860f74c2c4663f131d177a1e408db672d11ee86648ae3587702480ecf74fc6c"
Jul 28 02:45:47.539176 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.539076    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2860f74c2c4663f131d177a1e408db672d11ee86648ae3587702480ecf74fc6c"} err="failed to get container status \"2860f74c2c4663f131d177a1e408db672d11ee86648ae3587702480ecf74fc6c\": rpc error: code = NotFound desc = could not find container \"2860f74c2c4663f131d177a1e408db672d11ee86648ae3587702480ecf74fc6c\": container with ID starting with 2860f74c2c4663f131d177a1e408db672d11ee86648ae3587702480ecf74fc6c not found: ID does not exist"
Jul 28 02:45:47.539176 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.539091    2555 scope.go:117] "RemoveContainer" containerID="eb36ff4ae5cef55e69fb07c67b6df6c9b54fc82a99db6be1babaf09c17d140c9"
Jul 28 02:45:47.539453 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:45:47.539431    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb36ff4ae5cef55e69fb07c67b6df6c9b54fc82a99db6be1babaf09c17d140c9\": container with ID starting with eb36ff4ae5cef55e69fb07c67b6df6c9b54fc82a99db6be1babaf09c17d140c9 not found: ID does not exist" containerID="eb36ff4ae5cef55e69fb07c67b6df6c9b54fc82a99db6be1babaf09c17d140c9"
Jul 28 02:45:47.539515 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.539457    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb36ff4ae5cef55e69fb07c67b6df6c9b54fc82a99db6be1babaf09c17d140c9"} err="failed to get container status \"eb36ff4ae5cef55e69fb07c67b6df6c9b54fc82a99db6be1babaf09c17d140c9\": rpc error: code = NotFound desc = could not find container \"eb36ff4ae5cef55e69fb07c67b6df6c9b54fc82a99db6be1babaf09c17d140c9\": container with ID starting with eb36ff4ae5cef55e69fb07c67b6df6c9b54fc82a99db6be1babaf09c17d140c9 not found: ID does not exist"
Jul 28 02:45:47.545434 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.545413    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"]
Jul 28 02:45:47.551626 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:47.551600    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bszzz"]
Jul 28 02:45:49.451727 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:49.451693    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0632e025-c6b0-4a6a-92c4-9e3c793ed001" path="/var/lib/kubelet/pods/0632e025-c6b0-4a6a-92c4-9e3c793ed001/volumes"
Jul 28 02:45:50.529329 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:50.529297    2555 generic.go:356] "Generic (PLEG): container finished" podID="4782d67b-d390-49de-a187-6860617ea785" containerID="79b622450068b3af1406387edc13915e2f26049c6f79362b9baa58b4889681a7" exitCode=0
Jul 28 02:45:50.529707 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:50.529367    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" event={"ID":"4782d67b-d390-49de-a187-6860617ea785","Type":"ContainerDied","Data":"79b622450068b3af1406387edc13915e2f26049c6f79362b9baa58b4889681a7"}
Jul 28 02:45:51.535123 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:51.535088    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" event={"ID":"4782d67b-d390-49de-a187-6860617ea785","Type":"ContainerStarted","Data":"0b78892fca1cdcaafd606c77de2c2a29c6e56a8d1277c9c1a65d2068ac443a34"}
Jul 28 02:45:51.535552 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:51.535161    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" event={"ID":"4782d67b-d390-49de-a187-6860617ea785","Type":"ContainerStarted","Data":"e0a48c052605fc281b8a770d6101d80df98e78b97c1f0ab3b9d3536b8f5cefe6"}
Jul 28 02:45:51.535552 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:51.535493    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:51.557398 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:51.557343    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" podStartSLOduration=3.044031992 podStartE2EDuration="6.557327747s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.513295755s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:45:51.557284615 +0000 UTC m=+4798.566803040" watchObservedRunningTime="2026-07-28 02:45:51.557327747 +0000 UTC m=+4798.566846172"
Jul 28 02:45:52.539020 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:52.538967    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:52.540409 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:52.540375    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 28 02:45:53.542456 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:53.542415    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 28 02:45:58.546798 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:58.546770    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:45:58.547254 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:45:58.547229    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 28 02:46:08.547754 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:46:08.547717    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 28 02:46:18.547920 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:46:18.547876    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 28 02:46:28.547857 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:46:28.547816    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 28 02:46:38.547579 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:46:38.547517    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 28 02:46:48.548044 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:46:48.547959    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 28 02:46:58.548770 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:46:58.548736    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:47:05.862447 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:05.862413    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"]
Jul 28 02:47:05.862927 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:05.862686    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kserve-container" containerID="cri-o://e0a48c052605fc281b8a770d6101d80df98e78b97c1f0ab3b9d3536b8f5cefe6" gracePeriod=30
Jul 28 02:47:05.862927 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:05.862703    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kube-rbac-proxy" containerID="cri-o://0b78892fca1cdcaafd606c77de2c2a29c6e56a8d1277c9c1a65d2068ac443a34" gracePeriod=30
Jul 28 02:47:06.794165 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:06.793368    2555 generic.go:356] "Generic (PLEG): container finished" podID="4782d67b-d390-49de-a187-6860617ea785" containerID="0b78892fca1cdcaafd606c77de2c2a29c6e56a8d1277c9c1a65d2068ac443a34" exitCode=2
Jul 28 02:47:06.794165 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:06.793476    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" event={"ID":"4782d67b-d390-49de-a187-6860617ea785","Type":"ContainerDied","Data":"0b78892fca1cdcaafd606c77de2c2a29c6e56a8d1277c9c1a65d2068ac443a34"}
Jul 28 02:47:08.542805 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:08.542763    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Jul 28 02:47:08.547890 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:08.547860    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 28 02:47:09.498636 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.498600    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:47:09.562900 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.562866    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hrtr\" (UniqueName: \"kubernetes.io/projected/4782d67b-d390-49de-a187-6860617ea785-kube-api-access-9hrtr\") pod \"4782d67b-d390-49de-a187-6860617ea785\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") "
Jul 28 02:47:09.562900 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.562908    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4782d67b-d390-49de-a187-6860617ea785-proxy-tls\") pod \"4782d67b-d390-49de-a187-6860617ea785\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") "
Jul 28 02:47:09.563394 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.562947    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4782d67b-d390-49de-a187-6860617ea785-kserve-provision-location\") pod \"4782d67b-d390-49de-a187-6860617ea785\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") "
Jul 28 02:47:09.563394 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.562966    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4782d67b-d390-49de-a187-6860617ea785-cabundle-cert\") pod \"4782d67b-d390-49de-a187-6860617ea785\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") "
Jul 28 02:47:09.563394 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.563004    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4782d67b-d390-49de-a187-6860617ea785-kube-rbac-proxy-sar-config\") pod \"4782d67b-d390-49de-a187-6860617ea785\" (UID: \"4782d67b-d390-49de-a187-6860617ea785\") "
Jul 28 02:47:09.563394 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.563311    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4782d67b-d390-49de-a187-6860617ea785-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4782d67b-d390-49de-a187-6860617ea785" (UID: "4782d67b-d390-49de-a187-6860617ea785"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:47:09.563394 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.563341    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4782d67b-d390-49de-a187-6860617ea785-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4782d67b-d390-49de-a187-6860617ea785" (UID: "4782d67b-d390-49de-a187-6860617ea785"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:47:09.563642 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.563437    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4782d67b-d390-49de-a187-6860617ea785-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4782d67b-d390-49de-a187-6860617ea785" (UID: "4782d67b-d390-49de-a187-6860617ea785"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:47:09.565099 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.565070    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4782d67b-d390-49de-a187-6860617ea785-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4782d67b-d390-49de-a187-6860617ea785" (UID: "4782d67b-d390-49de-a187-6860617ea785"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:47:09.565213 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.565183    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4782d67b-d390-49de-a187-6860617ea785-kube-api-access-9hrtr" (OuterVolumeSpecName: "kube-api-access-9hrtr") pod "4782d67b-d390-49de-a187-6860617ea785" (UID: "4782d67b-d390-49de-a187-6860617ea785"). InnerVolumeSpecName "kube-api-access-9hrtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:47:09.663465 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.663381    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hrtr\" (UniqueName: \"kubernetes.io/projected/4782d67b-d390-49de-a187-6860617ea785-kube-api-access-9hrtr\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:47:09.663465 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.663410    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4782d67b-d390-49de-a187-6860617ea785-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:47:09.663465 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.663421    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4782d67b-d390-49de-a187-6860617ea785-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:47:09.663465 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.663430    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4782d67b-d390-49de-a187-6860617ea785-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:47:09.663465 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.663438    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4782d67b-d390-49de-a187-6860617ea785-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:47:09.805605 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.805569    2555 generic.go:356] "Generic (PLEG): container finished" podID="4782d67b-d390-49de-a187-6860617ea785" containerID="e0a48c052605fc281b8a770d6101d80df98e78b97c1f0ab3b9d3536b8f5cefe6" exitCode=0
Jul 28 02:47:09.805794 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.805614    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" event={"ID":"4782d67b-d390-49de-a187-6860617ea785","Type":"ContainerDied","Data":"e0a48c052605fc281b8a770d6101d80df98e78b97c1f0ab3b9d3536b8f5cefe6"}
Jul 28 02:47:09.805794 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.805628    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"
Jul 28 02:47:09.805794 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.805672    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh" event={"ID":"4782d67b-d390-49de-a187-6860617ea785","Type":"ContainerDied","Data":"83c0fbefbe02f4c759ba4ecd574e66ec03277813bfa794a662855f3f97e0d579"}
Jul 28 02:47:09.805794 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.805708    2555 scope.go:117] "RemoveContainer" containerID="0b78892fca1cdcaafd606c77de2c2a29c6e56a8d1277c9c1a65d2068ac443a34"
Jul 28 02:47:09.814511 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.814491    2555 scope.go:117] "RemoveContainer" containerID="e0a48c052605fc281b8a770d6101d80df98e78b97c1f0ab3b9d3536b8f5cefe6"
Jul 28 02:47:09.822088 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.822068    2555 scope.go:117] "RemoveContainer" containerID="79b622450068b3af1406387edc13915e2f26049c6f79362b9baa58b4889681a7"
Jul 28 02:47:09.827796 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.827770    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"]
Jul 28 02:47:09.829536 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.829493    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v87jh"]
Jul 28 02:47:09.830461 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.830440    2555 scope.go:117] "RemoveContainer" containerID="0b78892fca1cdcaafd606c77de2c2a29c6e56a8d1277c9c1a65d2068ac443a34"
Jul 28 02:47:09.830834 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:47:09.830816    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b78892fca1cdcaafd606c77de2c2a29c6e56a8d1277c9c1a65d2068ac443a34\": container with ID starting with 0b78892fca1cdcaafd606c77de2c2a29c6e56a8d1277c9c1a65d2068ac443a34 not found: ID does not exist" containerID="0b78892fca1cdcaafd606c77de2c2a29c6e56a8d1277c9c1a65d2068ac443a34"
Jul 28 02:47:09.830892 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.830843    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b78892fca1cdcaafd606c77de2c2a29c6e56a8d1277c9c1a65d2068ac443a34"} err="failed to get container status \"0b78892fca1cdcaafd606c77de2c2a29c6e56a8d1277c9c1a65d2068ac443a34\": rpc error: code = NotFound desc = could not find container \"0b78892fca1cdcaafd606c77de2c2a29c6e56a8d1277c9c1a65d2068ac443a34\": container with ID starting with 0b78892fca1cdcaafd606c77de2c2a29c6e56a8d1277c9c1a65d2068ac443a34 not found: ID does not exist"
Jul 28 02:47:09.830892 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.830861    2555 scope.go:117] "RemoveContainer" containerID="e0a48c052605fc281b8a770d6101d80df98e78b97c1f0ab3b9d3536b8f5cefe6"
Jul 28 02:47:09.831108 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:47:09.831093    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a48c052605fc281b8a770d6101d80df98e78b97c1f0ab3b9d3536b8f5cefe6\": container with ID starting with e0a48c052605fc281b8a770d6101d80df98e78b97c1f0ab3b9d3536b8f5cefe6 not found: ID does not exist" containerID="e0a48c052605fc281b8a770d6101d80df98e78b97c1f0ab3b9d3536b8f5cefe6"
Jul 28 02:47:09.831156 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.831110    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a48c052605fc281b8a770d6101d80df98e78b97c1f0ab3b9d3536b8f5cefe6"} err="failed to get container status \"e0a48c052605fc281b8a770d6101d80df98e78b97c1f0ab3b9d3536b8f5cefe6\": rpc error: code = NotFound desc = could not find container \"e0a48c052605fc281b8a770d6101d80df98e78b97c1f0ab3b9d3536b8f5cefe6\": container with ID starting with e0a48c052605fc281b8a770d6101d80df98e78b97c1f0ab3b9d3536b8f5cefe6 not found: ID does not exist"
Jul 28 02:47:09.831156 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.831122    2555 scope.go:117] "RemoveContainer" containerID="79b622450068b3af1406387edc13915e2f26049c6f79362b9baa58b4889681a7"
Jul 28 02:47:09.831329 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:47:09.831309    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b622450068b3af1406387edc13915e2f26049c6f79362b9baa58b4889681a7\": container with ID starting with 79b622450068b3af1406387edc13915e2f26049c6f79362b9baa58b4889681a7 not found: ID does not exist" containerID="79b622450068b3af1406387edc13915e2f26049c6f79362b9baa58b4889681a7"
Jul 28 02:47:09.831371 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:09.831332    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b622450068b3af1406387edc13915e2f26049c6f79362b9baa58b4889681a7"} err="failed to get container status \"79b622450068b3af1406387edc13915e2f26049c6f79362b9baa58b4889681a7\": rpc error: code = NotFound desc = could not find container \"79b622450068b3af1406387edc13915e2f26049c6f79362b9baa58b4889681a7\": container with ID starting with 79b622450068b3af1406387edc13915e2f26049c6f79362b9baa58b4889681a7 not found: ID does not exist"
Jul 28 02:47:11.033181 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033141    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"]
Jul 28 02:47:11.033590 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033432    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="storage-initializer"
Jul 28 02:47:11.033590 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033444    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="storage-initializer"
Jul 28 02:47:11.033590 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033455    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0632e025-c6b0-4a6a-92c4-9e3c793ed001" containerName="storage-initializer"
Jul 28 02:47:11.033590 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033461    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0632e025-c6b0-4a6a-92c4-9e3c793ed001" containerName="storage-initializer"
Jul 28 02:47:11.033590 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033470    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kube-rbac-proxy"
Jul 28 02:47:11.033590 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033475    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kube-rbac-proxy"
Jul 28 02:47:11.033590 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033481    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0632e025-c6b0-4a6a-92c4-9e3c793ed001" containerName="kube-rbac-proxy"
Jul 28 02:47:11.033590 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033486    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0632e025-c6b0-4a6a-92c4-9e3c793ed001" containerName="kube-rbac-proxy"
Jul 28 02:47:11.033590 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033498    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kserve-container"
Jul 28 02:47:11.033590 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033503    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kserve-container"
Jul 28 02:47:11.033590 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033509    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0632e025-c6b0-4a6a-92c4-9e3c793ed001" containerName="kserve-container"
Jul 28 02:47:11.033590 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033514    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0632e025-c6b0-4a6a-92c4-9e3c793ed001" containerName="kserve-container"
Jul 28 02:47:11.033590 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033571    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0632e025-c6b0-4a6a-92c4-9e3c793ed001" containerName="kserve-container"
Jul 28 02:47:11.033590 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033580    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0632e025-c6b0-4a6a-92c4-9e3c793ed001" containerName="kube-rbac-proxy"
Jul 28 02:47:11.033590 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033586    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kube-rbac-proxy"
Jul 28 02:47:11.033590 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.033592    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4782d67b-d390-49de-a187-6860617ea785" containerName="kserve-container"
Jul 28 02:47:11.037906 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.037887    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.040545 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.040505    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 28 02:47:11.040657 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.040510    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 02:47:11.040844 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.040828    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:47:11.040928 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.040871    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:47:11.040928 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.040827    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:47:11.041626 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.041611    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:47:11.041884 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.041870    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 02:47:11.046604 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.046577    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"]
Jul 28 02:47:11.175909 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.175871    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d000259a-8647-4075-8993-a3531b4ba8ba-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.175909 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.175907    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q92l5\" (UniqueName: \"kubernetes.io/projected/d000259a-8647-4075-8993-a3531b4ba8ba-kube-api-access-q92l5\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.176118 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.175955    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d000259a-8647-4075-8993-a3531b4ba8ba-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.176118 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.175982    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d000259a-8647-4075-8993-a3531b4ba8ba-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.176118 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.176012    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d000259a-8647-4075-8993-a3531b4ba8ba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.276778 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.276729    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d000259a-8647-4075-8993-a3531b4ba8ba-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.276778 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.276786    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q92l5\" (UniqueName: \"kubernetes.io/projected/d000259a-8647-4075-8993-a3531b4ba8ba-kube-api-access-q92l5\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.277050 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.276836    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d000259a-8647-4075-8993-a3531b4ba8ba-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.277050 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.276871    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d000259a-8647-4075-8993-a3531b4ba8ba-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.277050 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:47:11.276882    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Jul 28 02:47:11.277050 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.276900    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d000259a-8647-4075-8993-a3531b4ba8ba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.277050 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:47:11.276956    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d000259a-8647-4075-8993-a3531b4ba8ba-proxy-tls podName:d000259a-8647-4075-8993-a3531b4ba8ba nodeName:}" failed. No retries permitted until 2026-07-28 02:47:11.776934223 +0000 UTC m=+4878.786452627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d000259a-8647-4075-8993-a3531b4ba8ba-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5" (UID: "d000259a-8647-4075-8993-a3531b4ba8ba") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Jul 28 02:47:11.277393 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.277230    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d000259a-8647-4075-8993-a3531b4ba8ba-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.277515 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.277495    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d000259a-8647-4075-8993-a3531b4ba8ba-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.277655 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.277636    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d000259a-8647-4075-8993-a3531b4ba8ba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.286171 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.286111    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q92l5\" (UniqueName: \"kubernetes.io/projected/d000259a-8647-4075-8993-a3531b4ba8ba-kube-api-access-q92l5\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.451070 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.451033    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4782d67b-d390-49de-a187-6860617ea785" path="/var/lib/kubelet/pods/4782d67b-d390-49de-a187-6860617ea785/volumes"
Jul 28 02:47:11.781374 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.781340    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d000259a-8647-4075-8993-a3531b4ba8ba-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.783897 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.783864    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d000259a-8647-4075-8993-a3531b4ba8ba-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:11.948261 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:11.948202    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:12.081670 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:12.081585    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"]
Jul 28 02:47:12.084325 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:47:12.084299    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd000259a_8647_4075_8993_a3531b4ba8ba.slice/crio-576b87a5e742e79e38c0274c87a752963ad7cf5f02b9ced92c8d05820437872d WatchSource:0}: Error finding container 576b87a5e742e79e38c0274c87a752963ad7cf5f02b9ced92c8d05820437872d: Status 404 returned error can't find the container with id 576b87a5e742e79e38c0274c87a752963ad7cf5f02b9ced92c8d05820437872d
Jul 28 02:47:12.818609 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:12.818569    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5" event={"ID":"d000259a-8647-4075-8993-a3531b4ba8ba","Type":"ContainerStarted","Data":"ec46f3dac05980d4c899ba360e9da9af5f48e8e255139ea6b0cf6cef9bb45569"}
Jul 28 02:47:12.818779 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:12.818636    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5" event={"ID":"d000259a-8647-4075-8993-a3531b4ba8ba","Type":"ContainerStarted","Data":"576b87a5e742e79e38c0274c87a752963ad7cf5f02b9ced92c8d05820437872d"}
Jul 28 02:47:16.833909 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:16.833872    2555 generic.go:356] "Generic (PLEG): container finished" podID="d000259a-8647-4075-8993-a3531b4ba8ba" containerID="ec46f3dac05980d4c899ba360e9da9af5f48e8e255139ea6b0cf6cef9bb45569" exitCode=0
Jul 28 02:47:16.834332 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:16.833944    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5" event={"ID":"d000259a-8647-4075-8993-a3531b4ba8ba","Type":"ContainerDied","Data":"ec46f3dac05980d4c899ba360e9da9af5f48e8e255139ea6b0cf6cef9bb45569"}
Jul 28 02:47:17.838972 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:17.838936    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5" event={"ID":"d000259a-8647-4075-8993-a3531b4ba8ba","Type":"ContainerStarted","Data":"637272fed3940179c1db6e8ca89b2f3af8fdbdbb89c016192cb000e3cfdefce9"}
Jul 28 02:47:17.839444 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:17.839000    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5" event={"ID":"d000259a-8647-4075-8993-a3531b4ba8ba","Type":"ContainerStarted","Data":"5d4afeaf24ee26b196a1e156af4259f5fa5ca1a0efc80cbfe8f020acc3228520"}
Jul 28 02:47:17.839444 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:17.839210    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:17.839444 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:17.839253    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:17.859659 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:17.859608    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5" podStartSLOduration=2.992016089 podStartE2EDuration="6.859594142s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.867578053s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:47:17.858599089 +0000 UTC m=+4884.868117507" watchObservedRunningTime="2026-07-28 02:47:17.859594142 +0000 UTC m=+4884.869112568"
Jul 28 02:47:23.847372 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:23.847345    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:47:53.858801 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:47:53.858750    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5" podUID="d000259a-8647-4075-8993-a3531b4ba8ba" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 02:48:03.849818 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:03.849786    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:48:11.140583 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:11.140545    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"]
Jul 28 02:48:11.141053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:11.140826    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5" podUID="d000259a-8647-4075-8993-a3531b4ba8ba" containerName="kserve-container" containerID="cri-o://5d4afeaf24ee26b196a1e156af4259f5fa5ca1a0efc80cbfe8f020acc3228520" gracePeriod=30
Jul 28 02:48:11.141053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:11.140911    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5" podUID="d000259a-8647-4075-8993-a3531b4ba8ba" containerName="kube-rbac-proxy" containerID="cri-o://637272fed3940179c1db6e8ca89b2f3af8fdbdbb89c016192cb000e3cfdefce9" gracePeriod=30
Jul 28 02:48:12.021591 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:12.021551    2555 generic.go:356] "Generic (PLEG): container finished" podID="d000259a-8647-4075-8993-a3531b4ba8ba" containerID="637272fed3940179c1db6e8ca89b2f3af8fdbdbb89c016192cb000e3cfdefce9" exitCode=2
Jul 28 02:48:12.021789 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:12.021631    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5" event={"ID":"d000259a-8647-4075-8993-a3531b4ba8ba","Type":"ContainerDied","Data":"637272fed3940179c1db6e8ca89b2f3af8fdbdbb89c016192cb000e3cfdefce9"}
Jul 28 02:48:13.843121 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:13.843079    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5" podUID="d000259a-8647-4075-8993-a3531b4ba8ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.66:8643/healthz\": dial tcp 10.133.0.66:8643: connect: connection refused"
Jul 28 02:48:16.293402 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.293317    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"]
Jul 28 02:48:16.296728 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.296706    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:16.299440 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.299411    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jul 28 02:48:16.299440 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.299427    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 28 02:48:16.313613 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.313579    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"]
Jul 28 02:48:16.355370 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.355336    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxkz9\" (UniqueName: \"kubernetes.io/projected/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kube-api-access-bxkz9\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2vlcf\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:16.355370 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.355377    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2vlcf\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:16.355686 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.355402    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2vlcf\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:16.355686 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.355422    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2vlcf\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:16.355686 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.355491    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2vlcf\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:16.455978 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.455936    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2vlcf\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:16.456185 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.455998    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxkz9\" (UniqueName: \"kubernetes.io/projected/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kube-api-access-bxkz9\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2vlcf\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:16.456185 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.456037    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2vlcf\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:16.456185 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.456070    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2vlcf\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:16.456185 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.456098    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2vlcf\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:16.456395 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:48:16.456203    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 28 02:48:16.456395 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:48:16.456270    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-proxy-tls podName:a9524b04-b6a6-43a6-b3d3-6bbd617de41b nodeName:}" failed. No retries permitted until 2026-07-28 02:48:16.956249454 +0000 UTC m=+4943.965767862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-2vlcf" (UID: "a9524b04-b6a6-43a6-b3d3-6bbd617de41b") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 28 02:48:16.456559 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.456512    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2vlcf\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:16.456802 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.456782    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2vlcf\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:16.456875 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.456849    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2vlcf\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:16.465566 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.465516    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxkz9\" (UniqueName: \"kubernetes.io/projected/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kube-api-access-bxkz9\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2vlcf\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:16.959001 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.958962    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2vlcf\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:16.961792 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:16.961762    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2vlcf\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:17.206973 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:17.206922    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:17.338082 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:17.338047    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"]
Jul 28 02:48:17.341166 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:48:17.341140    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9524b04_b6a6_43a6_b3d3_6bbd617de41b.slice/crio-9986491d91e6a82251ea6070bca7634aa30679c77f8a37b70879e2ba48ccdce7 WatchSource:0}: Error finding container 9986491d91e6a82251ea6070bca7634aa30679c77f8a37b70879e2ba48ccdce7: Status 404 returned error can't find the container with id 9986491d91e6a82251ea6070bca7634aa30679c77f8a37b70879e2ba48ccdce7
Jul 28 02:48:18.042199 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:18.042162    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" event={"ID":"a9524b04-b6a6-43a6-b3d3-6bbd617de41b","Type":"ContainerStarted","Data":"4c27a21cb85b8d58957197e502c2efad34ebb1aa2524aee503ccb75d653e7a78"}
Jul 28 02:48:18.042385 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:18.042234    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" event={"ID":"a9524b04-b6a6-43a6-b3d3-6bbd617de41b","Type":"ContainerStarted","Data":"9986491d91e6a82251ea6070bca7634aa30679c77f8a37b70879e2ba48ccdce7"}
Jul 28 02:48:18.883623 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:18.883591    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:48:18.976282 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:18.976193    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d000259a-8647-4075-8993-a3531b4ba8ba-kube-rbac-proxy-sar-config\") pod \"d000259a-8647-4075-8993-a3531b4ba8ba\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") "
Jul 28 02:48:18.976447 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:18.976284    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d000259a-8647-4075-8993-a3531b4ba8ba-kserve-provision-location\") pod \"d000259a-8647-4075-8993-a3531b4ba8ba\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") "
Jul 28 02:48:18.976447 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:18.976384    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q92l5\" (UniqueName: \"kubernetes.io/projected/d000259a-8647-4075-8993-a3531b4ba8ba-kube-api-access-q92l5\") pod \"d000259a-8647-4075-8993-a3531b4ba8ba\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") "
Jul 28 02:48:18.976447 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:18.976430    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d000259a-8647-4075-8993-a3531b4ba8ba-cabundle-cert\") pod \"d000259a-8647-4075-8993-a3531b4ba8ba\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") "
Jul 28 02:48:18.976667 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:18.976452    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d000259a-8647-4075-8993-a3531b4ba8ba-proxy-tls\") pod \"d000259a-8647-4075-8993-a3531b4ba8ba\" (UID: \"d000259a-8647-4075-8993-a3531b4ba8ba\") "
Jul 28 02:48:18.976667 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:18.976620    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d000259a-8647-4075-8993-a3531b4ba8ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d000259a-8647-4075-8993-a3531b4ba8ba" (UID: "d000259a-8647-4075-8993-a3531b4ba8ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:48:18.976776 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:18.976654    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d000259a-8647-4075-8993-a3531b4ba8ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d000259a-8647-4075-8993-a3531b4ba8ba" (UID: "d000259a-8647-4075-8993-a3531b4ba8ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:48:18.976834 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:18.976816    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d000259a-8647-4075-8993-a3531b4ba8ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:48:18.976834 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:18.976831    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d000259a-8647-4075-8993-a3531b4ba8ba-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:48:18.976917 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:18.976834    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d000259a-8647-4075-8993-a3531b4ba8ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d000259a-8647-4075-8993-a3531b4ba8ba" (UID: "d000259a-8647-4075-8993-a3531b4ba8ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:48:18.978756 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:18.978732    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d000259a-8647-4075-8993-a3531b4ba8ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d000259a-8647-4075-8993-a3531b4ba8ba" (UID: "d000259a-8647-4075-8993-a3531b4ba8ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:48:18.978756 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:18.978742    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d000259a-8647-4075-8993-a3531b4ba8ba-kube-api-access-q92l5" (OuterVolumeSpecName: "kube-api-access-q92l5") pod "d000259a-8647-4075-8993-a3531b4ba8ba" (UID: "d000259a-8647-4075-8993-a3531b4ba8ba"). InnerVolumeSpecName "kube-api-access-q92l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:48:19.047635 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.047599    2555 generic.go:356] "Generic (PLEG): container finished" podID="d000259a-8647-4075-8993-a3531b4ba8ba" containerID="5d4afeaf24ee26b196a1e156af4259f5fa5ca1a0efc80cbfe8f020acc3228520" exitCode=0
Jul 28 02:48:19.047831 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.047668    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"
Jul 28 02:48:19.047831 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.047688    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5" event={"ID":"d000259a-8647-4075-8993-a3531b4ba8ba","Type":"ContainerDied","Data":"5d4afeaf24ee26b196a1e156af4259f5fa5ca1a0efc80cbfe8f020acc3228520"}
Jul 28 02:48:19.047831 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.047754    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5" event={"ID":"d000259a-8647-4075-8993-a3531b4ba8ba","Type":"ContainerDied","Data":"576b87a5e742e79e38c0274c87a752963ad7cf5f02b9ced92c8d05820437872d"}
Jul 28 02:48:19.047831 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.047781    2555 scope.go:117] "RemoveContainer" containerID="637272fed3940179c1db6e8ca89b2f3af8fdbdbb89c016192cb000e3cfdefce9"
Jul 28 02:48:19.056309 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.056291    2555 scope.go:117] "RemoveContainer" containerID="5d4afeaf24ee26b196a1e156af4259f5fa5ca1a0efc80cbfe8f020acc3228520"
Jul 28 02:48:19.063788 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.063766    2555 scope.go:117] "RemoveContainer" containerID="ec46f3dac05980d4c899ba360e9da9af5f48e8e255139ea6b0cf6cef9bb45569"
Jul 28 02:48:19.069756 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.069721    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"]
Jul 28 02:48:19.071106 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.071089    2555 scope.go:117] "RemoveContainer" containerID="637272fed3940179c1db6e8ca89b2f3af8fdbdbb89c016192cb000e3cfdefce9"
Jul 28 02:48:19.071413 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:48:19.071394    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637272fed3940179c1db6e8ca89b2f3af8fdbdbb89c016192cb000e3cfdefce9\": container with ID starting with 637272fed3940179c1db6e8ca89b2f3af8fdbdbb89c016192cb000e3cfdefce9 not found: ID does not exist" containerID="637272fed3940179c1db6e8ca89b2f3af8fdbdbb89c016192cb000e3cfdefce9"
Jul 28 02:48:19.071466 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.071420    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637272fed3940179c1db6e8ca89b2f3af8fdbdbb89c016192cb000e3cfdefce9"} err="failed to get container status \"637272fed3940179c1db6e8ca89b2f3af8fdbdbb89c016192cb000e3cfdefce9\": rpc error: code = NotFound desc = could not find container \"637272fed3940179c1db6e8ca89b2f3af8fdbdbb89c016192cb000e3cfdefce9\": container with ID starting with 637272fed3940179c1db6e8ca89b2f3af8fdbdbb89c016192cb000e3cfdefce9 not found: ID does not exist"
Jul 28 02:48:19.071466 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.071445    2555 scope.go:117] "RemoveContainer" containerID="5d4afeaf24ee26b196a1e156af4259f5fa5ca1a0efc80cbfe8f020acc3228520"
Jul 28 02:48:19.071724 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:48:19.071703    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4afeaf24ee26b196a1e156af4259f5fa5ca1a0efc80cbfe8f020acc3228520\": container with ID starting with 5d4afeaf24ee26b196a1e156af4259f5fa5ca1a0efc80cbfe8f020acc3228520 not found: ID does not exist" containerID="5d4afeaf24ee26b196a1e156af4259f5fa5ca1a0efc80cbfe8f020acc3228520"
Jul 28 02:48:19.071786 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.071734    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4afeaf24ee26b196a1e156af4259f5fa5ca1a0efc80cbfe8f020acc3228520"} err="failed to get container status \"5d4afeaf24ee26b196a1e156af4259f5fa5ca1a0efc80cbfe8f020acc3228520\": rpc error: code = NotFound desc = could not find container \"5d4afeaf24ee26b196a1e156af4259f5fa5ca1a0efc80cbfe8f020acc3228520\": container with ID starting with 5d4afeaf24ee26b196a1e156af4259f5fa5ca1a0efc80cbfe8f020acc3228520 not found: ID does not exist"
Jul 28 02:48:19.071786 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.071757    2555 scope.go:117] "RemoveContainer" containerID="ec46f3dac05980d4c899ba360e9da9af5f48e8e255139ea6b0cf6cef9bb45569"
Jul 28 02:48:19.071988 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:48:19.071970    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec46f3dac05980d4c899ba360e9da9af5f48e8e255139ea6b0cf6cef9bb45569\": container with ID starting with ec46f3dac05980d4c899ba360e9da9af5f48e8e255139ea6b0cf6cef9bb45569 not found: ID does not exist" containerID="ec46f3dac05980d4c899ba360e9da9af5f48e8e255139ea6b0cf6cef9bb45569"
Jul 28 02:48:19.072033 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.071992    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec46f3dac05980d4c899ba360e9da9af5f48e8e255139ea6b0cf6cef9bb45569"} err="failed to get container status \"ec46f3dac05980d4c899ba360e9da9af5f48e8e255139ea6b0cf6cef9bb45569\": rpc error: code = NotFound desc = could not find container \"ec46f3dac05980d4c899ba360e9da9af5f48e8e255139ea6b0cf6cef9bb45569\": container with ID starting with ec46f3dac05980d4c899ba360e9da9af5f48e8e255139ea6b0cf6cef9bb45569 not found: ID does not exist"
Jul 28 02:48:19.076238 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.076215    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5"]
Jul 28 02:48:19.077990 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.077971    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q92l5\" (UniqueName: \"kubernetes.io/projected/d000259a-8647-4075-8993-a3531b4ba8ba-kube-api-access-q92l5\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:48:19.078053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.077996    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d000259a-8647-4075-8993-a3531b4ba8ba-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:48:19.078053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.078010    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d000259a-8647-4075-8993-a3531b4ba8ba-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:48:19.451833 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.451799    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d000259a-8647-4075-8993-a3531b4ba8ba" path="/var/lib/kubelet/pods/d000259a-8647-4075-8993-a3531b4ba8ba/volumes"
Jul 28 02:48:19.843289 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:19.843239    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-9kdw5" podUID="d000259a-8647-4075-8993-a3531b4ba8ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.66:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 02:48:22.062399 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:22.062368    2555 generic.go:356] "Generic (PLEG): container finished" podID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerID="4c27a21cb85b8d58957197e502c2efad34ebb1aa2524aee503ccb75d653e7a78" exitCode=0
Jul 28 02:48:22.062805 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:22.062431    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" event={"ID":"a9524b04-b6a6-43a6-b3d3-6bbd617de41b","Type":"ContainerDied","Data":"4c27a21cb85b8d58957197e502c2efad34ebb1aa2524aee503ccb75d653e7a78"}
Jul 28 02:48:23.068709 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:23.068676    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" event={"ID":"a9524b04-b6a6-43a6-b3d3-6bbd617de41b","Type":"ContainerStarted","Data":"3eeef9095cce3c9e1965f9ac069b79b5a9080390b30771aa58bc3cf0f5b63c55"}
Jul 28 02:48:23.069121 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:23.068739    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" event={"ID":"a9524b04-b6a6-43a6-b3d3-6bbd617de41b","Type":"ContainerStarted","Data":"6946331c684fa8ec60450a4741eb7906114c201275f7a7cce09f93d65b6a7c88"}
Jul 28 02:48:23.069121 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:23.069029    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:23.069202 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:23.069182    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:23.070218 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:23.070193    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 28 02:48:23.087133 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:23.087075    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" podStartSLOduration=3.186478955 podStartE2EDuration="7.087060715s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.90058176s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:48:23.086560866 +0000 UTC m=+4950.096079308" watchObservedRunningTime="2026-07-28 02:48:23.087060715 +0000 UTC m=+4950.096579140"
Jul 28 02:48:24.072430 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:24.072392    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 28 02:48:29.076573 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:29.076517    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:48:29.077145 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:29.077118    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 28 02:48:39.077419 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:39.077379    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 28 02:48:49.077676 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:49.077636    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 28 02:48:59.077763 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:48:59.077717    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 28 02:49:09.077690 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:09.077641    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 28 02:49:19.077843 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:19.077801    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 28 02:49:29.077843 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:29.077807    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:49:36.415423 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:36.415385    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"]
Jul 28 02:49:36.415885 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:36.415669    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kserve-container" containerID="cri-o://6946331c684fa8ec60450a4741eb7906114c201275f7a7cce09f93d65b6a7c88" gracePeriod=30
Jul 28 02:49:36.415885 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:36.415724    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kube-rbac-proxy" containerID="cri-o://3eeef9095cce3c9e1965f9ac069b79b5a9080390b30771aa58bc3cf0f5b63c55" gracePeriod=30
Jul 28 02:49:37.321894 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:37.321857    2555 generic.go:356] "Generic (PLEG): container finished" podID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerID="3eeef9095cce3c9e1965f9ac069b79b5a9080390b30771aa58bc3cf0f5b63c55" exitCode=2
Jul 28 02:49:37.322077 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:37.321917    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" event={"ID":"a9524b04-b6a6-43a6-b3d3-6bbd617de41b","Type":"ContainerDied","Data":"3eeef9095cce3c9e1965f9ac069b79b5a9080390b30771aa58bc3cf0f5b63c55"}
Jul 28 02:49:39.072851 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:39.072812    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.67:8643/healthz\": dial tcp 10.133.0.67:8643: connect: connection refused"
Jul 28 02:49:39.077128 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:39.077102    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 28 02:49:40.164509 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.164485    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:49:40.181698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.181673    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-cabundle-cert\") pod \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") "
Jul 28 02:49:40.181801 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.181714    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kube-rbac-proxy-sar-config\") pod \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") "
Jul 28 02:49:40.181801 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.181751    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxkz9\" (UniqueName: \"kubernetes.io/projected/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kube-api-access-bxkz9\") pod \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") "
Jul 28 02:49:40.181801 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.181772    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-proxy-tls\") pod \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") "
Jul 28 02:49:40.181945 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.181801    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kserve-provision-location\") pod \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\" (UID: \"a9524b04-b6a6-43a6-b3d3-6bbd617de41b\") "
Jul 28 02:49:40.182041 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.182007    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a9524b04-b6a6-43a6-b3d3-6bbd617de41b" (UID: "a9524b04-b6a6-43a6-b3d3-6bbd617de41b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:49:40.182209 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.182182    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a9524b04-b6a6-43a6-b3d3-6bbd617de41b" (UID: "a9524b04-b6a6-43a6-b3d3-6bbd617de41b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:49:40.182209 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.182197    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a9524b04-b6a6-43a6-b3d3-6bbd617de41b" (UID: "a9524b04-b6a6-43a6-b3d3-6bbd617de41b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:49:40.184609 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.184576    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kube-api-access-bxkz9" (OuterVolumeSpecName: "kube-api-access-bxkz9") pod "a9524b04-b6a6-43a6-b3d3-6bbd617de41b" (UID: "a9524b04-b6a6-43a6-b3d3-6bbd617de41b"). InnerVolumeSpecName "kube-api-access-bxkz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:49:40.184609 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.184595    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a9524b04-b6a6-43a6-b3d3-6bbd617de41b" (UID: "a9524b04-b6a6-43a6-b3d3-6bbd617de41b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:49:40.283241 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.283199    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:49:40.283241 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.283235    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:49:40.283241 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.283247    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxkz9\" (UniqueName: \"kubernetes.io/projected/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kube-api-access-bxkz9\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:49:40.283487 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.283256    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:49:40.283487 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.283266    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9524b04-b6a6-43a6-b3d3-6bbd617de41b-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:49:40.334182 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.334149    2555 generic.go:356] "Generic (PLEG): container finished" podID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerID="6946331c684fa8ec60450a4741eb7906114c201275f7a7cce09f93d65b6a7c88" exitCode=0
Jul 28 02:49:40.334354 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.334214    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"
Jul 28 02:49:40.334354 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.334216    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" event={"ID":"a9524b04-b6a6-43a6-b3d3-6bbd617de41b","Type":"ContainerDied","Data":"6946331c684fa8ec60450a4741eb7906114c201275f7a7cce09f93d65b6a7c88"}
Jul 28 02:49:40.334354 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.334340    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf" event={"ID":"a9524b04-b6a6-43a6-b3d3-6bbd617de41b","Type":"ContainerDied","Data":"9986491d91e6a82251ea6070bca7634aa30679c77f8a37b70879e2ba48ccdce7"}
Jul 28 02:49:40.334515 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.334365    2555 scope.go:117] "RemoveContainer" containerID="3eeef9095cce3c9e1965f9ac069b79b5a9080390b30771aa58bc3cf0f5b63c55"
Jul 28 02:49:40.345131 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.345107    2555 scope.go:117] "RemoveContainer" containerID="6946331c684fa8ec60450a4741eb7906114c201275f7a7cce09f93d65b6a7c88"
Jul 28 02:49:40.352450 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.352432    2555 scope.go:117] "RemoveContainer" containerID="4c27a21cb85b8d58957197e502c2efad34ebb1aa2524aee503ccb75d653e7a78"
Jul 28 02:49:40.358830 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.358806    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"]
Jul 28 02:49:40.359938 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.359924    2555 scope.go:117] "RemoveContainer" containerID="3eeef9095cce3c9e1965f9ac069b79b5a9080390b30771aa58bc3cf0f5b63c55"
Jul 28 02:49:40.360217 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:49:40.360200    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eeef9095cce3c9e1965f9ac069b79b5a9080390b30771aa58bc3cf0f5b63c55\": container with ID starting with 3eeef9095cce3c9e1965f9ac069b79b5a9080390b30771aa58bc3cf0f5b63c55 not found: ID does not exist" containerID="3eeef9095cce3c9e1965f9ac069b79b5a9080390b30771aa58bc3cf0f5b63c55"
Jul 28 02:49:40.360259 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.360225    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eeef9095cce3c9e1965f9ac069b79b5a9080390b30771aa58bc3cf0f5b63c55"} err="failed to get container status \"3eeef9095cce3c9e1965f9ac069b79b5a9080390b30771aa58bc3cf0f5b63c55\": rpc error: code = NotFound desc = could not find container \"3eeef9095cce3c9e1965f9ac069b79b5a9080390b30771aa58bc3cf0f5b63c55\": container with ID starting with 3eeef9095cce3c9e1965f9ac069b79b5a9080390b30771aa58bc3cf0f5b63c55 not found: ID does not exist"
Jul 28 02:49:40.360259 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.360241    2555 scope.go:117] "RemoveContainer" containerID="6946331c684fa8ec60450a4741eb7906114c201275f7a7cce09f93d65b6a7c88"
Jul 28 02:49:40.360423 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:49:40.360408    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6946331c684fa8ec60450a4741eb7906114c201275f7a7cce09f93d65b6a7c88\": container with ID starting with 6946331c684fa8ec60450a4741eb7906114c201275f7a7cce09f93d65b6a7c88 not found: ID does not exist" containerID="6946331c684fa8ec60450a4741eb7906114c201275f7a7cce09f93d65b6a7c88"
Jul 28 02:49:40.360460 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.360427    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6946331c684fa8ec60450a4741eb7906114c201275f7a7cce09f93d65b6a7c88"} err="failed to get container status \"6946331c684fa8ec60450a4741eb7906114c201275f7a7cce09f93d65b6a7c88\": rpc error: code = NotFound desc = could not find container \"6946331c684fa8ec60450a4741eb7906114c201275f7a7cce09f93d65b6a7c88\": container with ID starting with 6946331c684fa8ec60450a4741eb7906114c201275f7a7cce09f93d65b6a7c88 not found: ID does not exist"
Jul 28 02:49:40.360460 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.360439    2555 scope.go:117] "RemoveContainer" containerID="4c27a21cb85b8d58957197e502c2efad34ebb1aa2524aee503ccb75d653e7a78"
Jul 28 02:49:40.360711 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:49:40.360691    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c27a21cb85b8d58957197e502c2efad34ebb1aa2524aee503ccb75d653e7a78\": container with ID starting with 4c27a21cb85b8d58957197e502c2efad34ebb1aa2524aee503ccb75d653e7a78 not found: ID does not exist" containerID="4c27a21cb85b8d58957197e502c2efad34ebb1aa2524aee503ccb75d653e7a78"
Jul 28 02:49:40.360761 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.360717    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c27a21cb85b8d58957197e502c2efad34ebb1aa2524aee503ccb75d653e7a78"} err="failed to get container status \"4c27a21cb85b8d58957197e502c2efad34ebb1aa2524aee503ccb75d653e7a78\": rpc error: code = NotFound desc = could not find container \"4c27a21cb85b8d58957197e502c2efad34ebb1aa2524aee503ccb75d653e7a78\": container with ID starting with 4c27a21cb85b8d58957197e502c2efad34ebb1aa2524aee503ccb75d653e7a78 not found: ID does not exist"
Jul 28 02:49:40.367337 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:40.367314    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2vlcf"]
Jul 28 02:49:41.451128 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.451096    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" path="/var/lib/kubelet/pods/a9524b04-b6a6-43a6-b3d3-6bbd617de41b/volumes"
Jul 28 02:49:41.568404 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568369    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"]
Jul 28 02:49:41.568639 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568626    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kserve-container"
Jul 28 02:49:41.568698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568641    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kserve-container"
Jul 28 02:49:41.568698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568651    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d000259a-8647-4075-8993-a3531b4ba8ba" containerName="storage-initializer"
Jul 28 02:49:41.568698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568656    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d000259a-8647-4075-8993-a3531b4ba8ba" containerName="storage-initializer"
Jul 28 02:49:41.568698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568665    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d000259a-8647-4075-8993-a3531b4ba8ba" containerName="kube-rbac-proxy"
Jul 28 02:49:41.568698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568670    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d000259a-8647-4075-8993-a3531b4ba8ba" containerName="kube-rbac-proxy"
Jul 28 02:49:41.568698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568676    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="storage-initializer"
Jul 28 02:49:41.568698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568681    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="storage-initializer"
Jul 28 02:49:41.568698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568691    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kube-rbac-proxy"
Jul 28 02:49:41.568698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568696    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kube-rbac-proxy"
Jul 28 02:49:41.568977 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568703    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d000259a-8647-4075-8993-a3531b4ba8ba" containerName="kserve-container"
Jul 28 02:49:41.568977 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568708    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d000259a-8647-4075-8993-a3531b4ba8ba" containerName="kserve-container"
Jul 28 02:49:41.568977 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568747    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d000259a-8647-4075-8993-a3531b4ba8ba" containerName="kserve-container"
Jul 28 02:49:41.568977 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568754    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kube-rbac-proxy"
Jul 28 02:49:41.568977 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568762    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d000259a-8647-4075-8993-a3531b4ba8ba" containerName="kube-rbac-proxy"
Jul 28 02:49:41.568977 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.568770    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9524b04-b6a6-43a6-b3d3-6bbd617de41b" containerName="kserve-container"
Jul 28 02:49:41.573391 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.573373    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:41.576561 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.576539    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:49:41.576561 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.576559    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jul 28 02:49:41.576912 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.576895    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 02:49:41.577025 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.576895    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jul 28 02:49:41.577025 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.576998    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:49:41.577201 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.577187    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:49:41.582474 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.582455    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"]
Jul 28 02:49:41.591610 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.591582    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-frpps\" (UID: \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:41.591734 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.591619    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-frpps\" (UID: \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:41.591734 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.591638    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz4zc\" (UniqueName: \"kubernetes.io/projected/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kube-api-access-dz4zc\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-frpps\" (UID: \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:41.591734 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.591707    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ac1ea67-9234-4855-a0c1-5dfb7832337b-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-frpps\" (UID: \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:41.692408 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.692364    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-frpps\" (UID: \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:41.692408 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.692408    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-frpps\" (UID: \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:41.692730 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.692426    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz4zc\" (UniqueName: \"kubernetes.io/projected/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kube-api-access-dz4zc\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-frpps\" (UID: \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:41.692730 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.692455    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ac1ea67-9234-4855-a0c1-5dfb7832337b-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-frpps\" (UID: \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:41.692923 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.692898    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-frpps\" (UID: \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:41.693343 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.693290    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-frpps\" (UID: \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:41.695326 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.695306    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ac1ea67-9234-4855-a0c1-5dfb7832337b-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-frpps\" (UID: \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:41.703546 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.703466    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz4zc\" (UniqueName: \"kubernetes.io/projected/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kube-api-access-dz4zc\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-frpps\" (UID: \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:41.884437 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:41.884398    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:42.012366 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:42.012335    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"]
Jul 28 02:49:42.015860 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:49:42.015828    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac1ea67_9234_4855_a0c1_5dfb7832337b.slice/crio-fbd55c8e342df92f3954c54115a97cd37889c74216b102b05ba707ce1d370910 WatchSource:0}: Error finding container fbd55c8e342df92f3954c54115a97cd37889c74216b102b05ba707ce1d370910: Status 404 returned error can't find the container with id fbd55c8e342df92f3954c54115a97cd37889c74216b102b05ba707ce1d370910
Jul 28 02:49:42.343508 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:42.343465    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" event={"ID":"5ac1ea67-9234-4855-a0c1-5dfb7832337b","Type":"ContainerStarted","Data":"c4c3fcbd406720630020c0f67a311f46b518ad6719ed5fce4f0c567de981c114"}
Jul 28 02:49:42.343692 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:42.343570    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" event={"ID":"5ac1ea67-9234-4855-a0c1-5dfb7832337b","Type":"ContainerStarted","Data":"fbd55c8e342df92f3954c54115a97cd37889c74216b102b05ba707ce1d370910"}
Jul 28 02:49:43.348159 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:43.348064    2555 generic.go:356] "Generic (PLEG): container finished" podID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerID="c4c3fcbd406720630020c0f67a311f46b518ad6719ed5fce4f0c567de981c114" exitCode=0
Jul 28 02:49:43.348581 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:43.348155    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" event={"ID":"5ac1ea67-9234-4855-a0c1-5dfb7832337b","Type":"ContainerDied","Data":"c4c3fcbd406720630020c0f67a311f46b518ad6719ed5fce4f0c567de981c114"}
Jul 28 02:49:44.353655 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:44.353569    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" event={"ID":"5ac1ea67-9234-4855-a0c1-5dfb7832337b","Type":"ContainerStarted","Data":"a554d11d7d602522b52f6e454a5e62af924ac0a2727fdf9132c13d9401397376"}
Jul 28 02:49:44.353655 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:44.353635    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" event={"ID":"5ac1ea67-9234-4855-a0c1-5dfb7832337b","Type":"ContainerStarted","Data":"9f0841ff70a24ee30f3a3ee84cdca63a1fc8d6c5b0cdccbd6baf675db4aa24b2"}
Jul 28 02:49:44.354065 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:44.353698    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:44.375169 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:44.375120    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" podStartSLOduration=2.39381269 podStartE2EDuration="3.375106158s" totalImagesPullingTime="0s" totalInitContainerRuntime="981.293468ms" isStatefulPod=false podCreationTimestamp="2026-07-28 02:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:49:44.373434234 +0000 UTC m=+5031.382952667" watchObservedRunningTime="2026-07-28 02:49:44.375106158 +0000 UTC m=+5031.384624583"
Jul 28 02:49:45.357785 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:45.357757    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:45.358877 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:45.358845    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 02:49:46.361122 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:46.361076    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 02:49:51.365271 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:51.365235    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:49:51.365878 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:49:51.365849    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 02:50:01.366841 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:50:01.366801    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 02:50:11.366170 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:50:11.366121    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 02:50:21.365984 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:50:21.365935    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 02:50:31.366620 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:50:31.366580    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 02:50:41.365792 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:50:41.365744    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 02:50:51.366224 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:50:51.366183    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 02:51:01.367220 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:01.367189    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:51:11.780977 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:11.780947    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"]
Jul 28 02:51:11.781370 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:11.781190    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kserve-container" containerID="cri-o://9f0841ff70a24ee30f3a3ee84cdca63a1fc8d6c5b0cdccbd6baf675db4aa24b2" gracePeriod=30
Jul 28 02:51:11.781370 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:11.781249    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kube-rbac-proxy" containerID="cri-o://a554d11d7d602522b52f6e454a5e62af924ac0a2727fdf9132c13d9401397376" gracePeriod=30
Jul 28 02:51:11.931624 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:11.931591    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"]
Jul 28 02:51:11.934700 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:11.934683    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:11.939243 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:11.939216    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 28 02:51:11.939368 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:11.939340    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 28 02:51:11.939781 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:11.939765    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:51:11.973903 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:11.973866    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"]
Jul 28 02:51:12.031752 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.031654    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12f94aca-3a07-4a07-8f3c-98b8b920563b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:12.031752 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.031706    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12f94aca-3a07-4a07-8f3c-98b8b920563b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:12.031971 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.031764    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl2kp\" (UniqueName: \"kubernetes.io/projected/12f94aca-3a07-4a07-8f3c-98b8b920563b-kube-api-access-dl2kp\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:12.031971 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.031804    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12f94aca-3a07-4a07-8f3c-98b8b920563b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:12.031971 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.031835    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12f94aca-3a07-4a07-8f3c-98b8b920563b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:12.132755 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.132716    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12f94aca-3a07-4a07-8f3c-98b8b920563b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:12.132949 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.132778    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12f94aca-3a07-4a07-8f3c-98b8b920563b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:12.132949 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.132820    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12f94aca-3a07-4a07-8f3c-98b8b920563b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:12.132949 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.132848    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2kp\" (UniqueName: \"kubernetes.io/projected/12f94aca-3a07-4a07-8f3c-98b8b920563b-kube-api-access-dl2kp\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:12.132949 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.132878    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12f94aca-3a07-4a07-8f3c-98b8b920563b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:12.133217 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.133187    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12f94aca-3a07-4a07-8f3c-98b8b920563b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:12.133508 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.133482    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12f94aca-3a07-4a07-8f3c-98b8b920563b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:12.133618 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.133551    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12f94aca-3a07-4a07-8f3c-98b8b920563b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:12.135583 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.135562    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12f94aca-3a07-4a07-8f3c-98b8b920563b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:12.142564 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.142542    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl2kp\" (UniqueName: \"kubernetes.io/projected/12f94aca-3a07-4a07-8f3c-98b8b920563b-kube-api-access-dl2kp\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:12.244687 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.244641    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:12.381201 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.381168    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"]
Jul 28 02:51:12.392409 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:51:12.392384    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f94aca_3a07_4a07_8f3c_98b8b920563b.slice/crio-c1f5476909cdd98cb2fd95eb586c7a8be4879b5c43ee08fea413b6170827a70b WatchSource:0}: Error finding container c1f5476909cdd98cb2fd95eb586c7a8be4879b5c43ee08fea413b6170827a70b: Status 404 returned error can't find the container with id c1f5476909cdd98cb2fd95eb586c7a8be4879b5c43ee08fea413b6170827a70b
Jul 28 02:51:12.653986 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.653883    2555 generic.go:356] "Generic (PLEG): container finished" podID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerID="a554d11d7d602522b52f6e454a5e62af924ac0a2727fdf9132c13d9401397376" exitCode=2
Jul 28 02:51:12.653986 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.653915    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" event={"ID":"5ac1ea67-9234-4855-a0c1-5dfb7832337b","Type":"ContainerDied","Data":"a554d11d7d602522b52f6e454a5e62af924ac0a2727fdf9132c13d9401397376"}
Jul 28 02:51:12.655628 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.655607    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" event={"ID":"12f94aca-3a07-4a07-8f3c-98b8b920563b","Type":"ContainerStarted","Data":"9a13248c3a52680799b5f9fc68e2a73c96f0d80e9f706d23c71c9235e6e3291a"}
Jul 28 02:51:12.655747 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:12.655658    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" event={"ID":"12f94aca-3a07-4a07-8f3c-98b8b920563b","Type":"ContainerStarted","Data":"c1f5476909cdd98cb2fd95eb586c7a8be4879b5c43ee08fea413b6170827a70b"}
Jul 28 02:51:13.660879 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:13.660797    2555 generic.go:356] "Generic (PLEG): container finished" podID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerID="9a13248c3a52680799b5f9fc68e2a73c96f0d80e9f706d23c71c9235e6e3291a" exitCode=0
Jul 28 02:51:13.660879 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:13.660864    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" event={"ID":"12f94aca-3a07-4a07-8f3c-98b8b920563b","Type":"ContainerDied","Data":"9a13248c3a52680799b5f9fc68e2a73c96f0d80e9f706d23c71c9235e6e3291a"}
Jul 28 02:51:14.667465 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:14.667377    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" event={"ID":"12f94aca-3a07-4a07-8f3c-98b8b920563b","Type":"ContainerStarted","Data":"de5c3f67a701bdd3761e8f3ac74f4d47433d0188086cca92a152e22634c39bef"}
Jul 28 02:51:14.667865 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:14.667478    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" event={"ID":"12f94aca-3a07-4a07-8f3c-98b8b920563b","Type":"ContainerStarted","Data":"b111324b026d6c817cc8e3320cd8181d577928291402fa28bd11ff71bdd2f89e"}
Jul 28 02:51:14.667865 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:14.667513    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:14.667865 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:14.667562    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:14.668860 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:14.668828    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 02:51:14.692932 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:14.692882    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" podStartSLOduration=2.765080551 podStartE2EDuration="3.692867864s" totalImagesPullingTime="0s" totalInitContainerRuntime="927.787313ms" isStatefulPod=false podCreationTimestamp="2026-07-28 02:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:51:14.69145136 +0000 UTC m=+5121.700969785" watchObservedRunningTime="2026-07-28 02:51:14.692867864 +0000 UTC m=+5121.702386289"
Jul 28 02:51:15.670778 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:15.670736    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 02:51:16.417805 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.417780    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:51:16.459666 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.459640    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kserve-provision-location\") pod \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\" (UID: \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\") "
Jul 28 02:51:16.459809 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.459681    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ac1ea67-9234-4855-a0c1-5dfb7832337b-proxy-tls\") pod \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\" (UID: \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\") "
Jul 28 02:51:16.459809 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.459707    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kube-rbac-proxy-sar-config\") pod \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\" (UID: \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\") "
Jul 28 02:51:16.459809 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.459743    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz4zc\" (UniqueName: \"kubernetes.io/projected/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kube-api-access-dz4zc\") pod \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\" (UID: \"5ac1ea67-9234-4855-a0c1-5dfb7832337b\") "
Jul 28 02:51:16.459992 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.459971    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ac1ea67-9234-4855-a0c1-5dfb7832337b" (UID: "5ac1ea67-9234-4855-a0c1-5dfb7832337b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:51:16.460073 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.460049    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ac1ea67-9234-4855-a0c1-5dfb7832337b" (UID: "5ac1ea67-9234-4855-a0c1-5dfb7832337b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:51:16.461887 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.461857    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac1ea67-9234-4855-a0c1-5dfb7832337b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ac1ea67-9234-4855-a0c1-5dfb7832337b" (UID: "5ac1ea67-9234-4855-a0c1-5dfb7832337b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:51:16.462017 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.461995    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kube-api-access-dz4zc" (OuterVolumeSpecName: "kube-api-access-dz4zc") pod "5ac1ea67-9234-4855-a0c1-5dfb7832337b" (UID: "5ac1ea67-9234-4855-a0c1-5dfb7832337b"). InnerVolumeSpecName "kube-api-access-dz4zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:51:16.560620 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.560493    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ac1ea67-9234-4855-a0c1-5dfb7832337b-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:51:16.560620 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.560570    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:51:16.560620 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.560588    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz4zc\" (UniqueName: \"kubernetes.io/projected/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kube-api-access-dz4zc\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:51:16.560620 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.560603    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac1ea67-9234-4855-a0c1-5dfb7832337b-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:51:16.675633 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.675598    2555 generic.go:356] "Generic (PLEG): container finished" podID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerID="9f0841ff70a24ee30f3a3ee84cdca63a1fc8d6c5b0cdccbd6baf675db4aa24b2" exitCode=0
Jul 28 02:51:16.676005 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.675656    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"
Jul 28 02:51:16.676005 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.675670    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" event={"ID":"5ac1ea67-9234-4855-a0c1-5dfb7832337b","Type":"ContainerDied","Data":"9f0841ff70a24ee30f3a3ee84cdca63a1fc8d6c5b0cdccbd6baf675db4aa24b2"}
Jul 28 02:51:16.676005 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.675719    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" event={"ID":"5ac1ea67-9234-4855-a0c1-5dfb7832337b","Type":"ContainerDied","Data":"fbd55c8e342df92f3954c54115a97cd37889c74216b102b05ba707ce1d370910"}
Jul 28 02:51:16.676005 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.675744    2555 scope.go:117] "RemoveContainer" containerID="a554d11d7d602522b52f6e454a5e62af924ac0a2727fdf9132c13d9401397376"
Jul 28 02:51:16.684492 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.684469    2555 scope.go:117] "RemoveContainer" containerID="9f0841ff70a24ee30f3a3ee84cdca63a1fc8d6c5b0cdccbd6baf675db4aa24b2"
Jul 28 02:51:16.691586 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.691569    2555 scope.go:117] "RemoveContainer" containerID="c4c3fcbd406720630020c0f67a311f46b518ad6719ed5fce4f0c567de981c114"
Jul 28 02:51:16.699118 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.699089    2555 scope.go:117] "RemoveContainer" containerID="a554d11d7d602522b52f6e454a5e62af924ac0a2727fdf9132c13d9401397376"
Jul 28 02:51:16.699258 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.699235    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"]
Jul 28 02:51:16.699385 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:51:16.699366    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a554d11d7d602522b52f6e454a5e62af924ac0a2727fdf9132c13d9401397376\": container with ID starting with a554d11d7d602522b52f6e454a5e62af924ac0a2727fdf9132c13d9401397376 not found: ID does not exist" containerID="a554d11d7d602522b52f6e454a5e62af924ac0a2727fdf9132c13d9401397376"
Jul 28 02:51:16.699453 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.699399    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a554d11d7d602522b52f6e454a5e62af924ac0a2727fdf9132c13d9401397376"} err="failed to get container status \"a554d11d7d602522b52f6e454a5e62af924ac0a2727fdf9132c13d9401397376\": rpc error: code = NotFound desc = could not find container \"a554d11d7d602522b52f6e454a5e62af924ac0a2727fdf9132c13d9401397376\": container with ID starting with a554d11d7d602522b52f6e454a5e62af924ac0a2727fdf9132c13d9401397376 not found: ID does not exist"
Jul 28 02:51:16.699453 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.699426    2555 scope.go:117] "RemoveContainer" containerID="9f0841ff70a24ee30f3a3ee84cdca63a1fc8d6c5b0cdccbd6baf675db4aa24b2"
Jul 28 02:51:16.699700 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:51:16.699685    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0841ff70a24ee30f3a3ee84cdca63a1fc8d6c5b0cdccbd6baf675db4aa24b2\": container with ID starting with 9f0841ff70a24ee30f3a3ee84cdca63a1fc8d6c5b0cdccbd6baf675db4aa24b2 not found: ID does not exist" containerID="9f0841ff70a24ee30f3a3ee84cdca63a1fc8d6c5b0cdccbd6baf675db4aa24b2"
Jul 28 02:51:16.699740 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.699705    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0841ff70a24ee30f3a3ee84cdca63a1fc8d6c5b0cdccbd6baf675db4aa24b2"} err="failed to get container status \"9f0841ff70a24ee30f3a3ee84cdca63a1fc8d6c5b0cdccbd6baf675db4aa24b2\": rpc error: code = NotFound desc = could not find container \"9f0841ff70a24ee30f3a3ee84cdca63a1fc8d6c5b0cdccbd6baf675db4aa24b2\": container with ID starting with 9f0841ff70a24ee30f3a3ee84cdca63a1fc8d6c5b0cdccbd6baf675db4aa24b2 not found: ID does not exist"
Jul 28 02:51:16.699740 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.699721    2555 scope.go:117] "RemoveContainer" containerID="c4c3fcbd406720630020c0f67a311f46b518ad6719ed5fce4f0c567de981c114"
Jul 28 02:51:16.699955 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:51:16.699934    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4c3fcbd406720630020c0f67a311f46b518ad6719ed5fce4f0c567de981c114\": container with ID starting with c4c3fcbd406720630020c0f67a311f46b518ad6719ed5fce4f0c567de981c114 not found: ID does not exist" containerID="c4c3fcbd406720630020c0f67a311f46b518ad6719ed5fce4f0c567de981c114"
Jul 28 02:51:16.700000 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.699962    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4c3fcbd406720630020c0f67a311f46b518ad6719ed5fce4f0c567de981c114"} err="failed to get container status \"c4c3fcbd406720630020c0f67a311f46b518ad6719ed5fce4f0c567de981c114\": rpc error: code = NotFound desc = could not find container \"c4c3fcbd406720630020c0f67a311f46b518ad6719ed5fce4f0c567de981c114\": container with ID starting with c4c3fcbd406720630020c0f67a311f46b518ad6719ed5fce4f0c567de981c114 not found: ID does not exist"
Jul 28 02:51:16.703931 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:16.703899    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps"]
Jul 28 02:51:17.361097 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:17.361058    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-frpps" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.68:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 02:51:17.451710 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:17.451672    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" path="/var/lib/kubelet/pods/5ac1ea67-9234-4855-a0c1-5dfb7832337b/volumes"
Jul 28 02:51:20.675691 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:20.675661    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:51:20.676263 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:20.676238    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 02:51:30.676539 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:30.676483    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 02:51:40.676760 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:40.676718    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 02:51:50.677172 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:51:50.677134    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 02:52:00.676428 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:00.676385    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 02:52:10.676479 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:10.676440    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 02:52:20.676780 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:20.676742    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 02:52:24.447093 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:24.447048    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 02:52:34.447723 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:34.447690    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:52:41.915317 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:41.915284    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"]
Jul 28 02:52:41.915920 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:41.915538    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kserve-container" containerID="cri-o://b111324b026d6c817cc8e3320cd8181d577928291402fa28bd11ff71bdd2f89e" gracePeriod=30
Jul 28 02:52:41.915920 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:41.915607    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kube-rbac-proxy" containerID="cri-o://de5c3f67a701bdd3761e8f3ac74f4d47433d0188086cca92a152e22634c39bef" gracePeriod=30
Jul 28 02:52:42.970696 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:42.970662    2555 generic.go:356] "Generic (PLEG): container finished" podID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerID="de5c3f67a701bdd3761e8f3ac74f4d47433d0188086cca92a152e22634c39bef" exitCode=2
Jul 28 02:52:42.971067 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:42.970731    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" event={"ID":"12f94aca-3a07-4a07-8f3c-98b8b920563b","Type":"ContainerDied","Data":"de5c3f67a701bdd3761e8f3ac74f4d47433d0188086cca92a152e22634c39bef"}
Jul 28 02:52:43.025033 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.024999    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"]
Jul 28 02:52:43.025276 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.025262    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kserve-container"
Jul 28 02:52:43.025320 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.025278    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kserve-container"
Jul 28 02:52:43.025320 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.025288    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kube-rbac-proxy"
Jul 28 02:52:43.025320 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.025294    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kube-rbac-proxy"
Jul 28 02:52:43.025320 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.025306    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="storage-initializer"
Jul 28 02:52:43.025320 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.025311    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="storage-initializer"
Jul 28 02:52:43.025477 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.025357    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kube-rbac-proxy"
Jul 28 02:52:43.025477 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.025365    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac1ea67-9234-4855-a0c1-5dfb7832337b" containerName="kserve-container"
Jul 28 02:52:43.028240 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.028220    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"
Jul 28 02:52:43.030978 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.030954    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jul 28 02:52:43.031063 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.030993    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jul 28 02:52:43.039556 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.039507    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"]
Jul 28 02:52:43.154666 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.154623    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af7e67d6-60c4-4ebc-bab7-543fe2991097-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b\" (UID: \"af7e67d6-60c4-4ebc-bab7-543fe2991097\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"
Jul 28 02:52:43.154854 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.154672    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af7e67d6-60c4-4ebc-bab7-543fe2991097-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b\" (UID: \"af7e67d6-60c4-4ebc-bab7-543fe2991097\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"
Jul 28 02:52:43.154854 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.154775    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af7e67d6-60c4-4ebc-bab7-543fe2991097-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b\" (UID: \"af7e67d6-60c4-4ebc-bab7-543fe2991097\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"
Jul 28 02:52:43.154956 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.154866    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5k8b\" (UniqueName: \"kubernetes.io/projected/af7e67d6-60c4-4ebc-bab7-543fe2991097-kube-api-access-x5k8b\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b\" (UID: \"af7e67d6-60c4-4ebc-bab7-543fe2991097\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"
Jul 28 02:52:43.255891 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.255854    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af7e67d6-60c4-4ebc-bab7-543fe2991097-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b\" (UID: \"af7e67d6-60c4-4ebc-bab7-543fe2991097\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"
Jul 28 02:52:43.256084 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.255907    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5k8b\" (UniqueName: \"kubernetes.io/projected/af7e67d6-60c4-4ebc-bab7-543fe2991097-kube-api-access-x5k8b\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b\" (UID: \"af7e67d6-60c4-4ebc-bab7-543fe2991097\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"
Jul 28 02:52:43.256084 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.255942    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af7e67d6-60c4-4ebc-bab7-543fe2991097-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b\" (UID: \"af7e67d6-60c4-4ebc-bab7-543fe2991097\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"
Jul 28 02:52:43.256084 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.255958    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af7e67d6-60c4-4ebc-bab7-543fe2991097-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b\" (UID: \"af7e67d6-60c4-4ebc-bab7-543fe2991097\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"
Jul 28 02:52:43.256318 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.256299    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af7e67d6-60c4-4ebc-bab7-543fe2991097-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b\" (UID: \"af7e67d6-60c4-4ebc-bab7-543fe2991097\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"
Jul 28 02:52:43.256721 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.256686    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af7e67d6-60c4-4ebc-bab7-543fe2991097-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b\" (UID: \"af7e67d6-60c4-4ebc-bab7-543fe2991097\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"
Jul 28 02:52:43.258651 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.258623    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af7e67d6-60c4-4ebc-bab7-543fe2991097-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b\" (UID: \"af7e67d6-60c4-4ebc-bab7-543fe2991097\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"
Jul 28 02:52:43.265110 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.265086    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5k8b\" (UniqueName: \"kubernetes.io/projected/af7e67d6-60c4-4ebc-bab7-543fe2991097-kube-api-access-x5k8b\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b\" (UID: \"af7e67d6-60c4-4ebc-bab7-543fe2991097\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"
Jul 28 02:52:43.338680 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.338624    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"
Jul 28 02:52:43.473886 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.473849    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"]
Jul 28 02:52:43.477079 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:52:43.477040    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf7e67d6_60c4_4ebc_bab7_543fe2991097.slice/crio-96f3a17077a296566b927cf1b8a4b0654601d3ff16ceb7a8fb5118be0d0b9215 WatchSource:0}: Error finding container 96f3a17077a296566b927cf1b8a4b0654601d3ff16ceb7a8fb5118be0d0b9215: Status 404 returned error can't find the container with id 96f3a17077a296566b927cf1b8a4b0654601d3ff16ceb7a8fb5118be0d0b9215
Jul 28 02:52:43.976055 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.975957    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b" event={"ID":"af7e67d6-60c4-4ebc-bab7-543fe2991097","Type":"ContainerStarted","Data":"5eb71a038174702fcdcee02a98102a4b72260a82c880e545d0c80ab1f1d6a6aa"}
Jul 28 02:52:43.976055 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:43.976042    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b" event={"ID":"af7e67d6-60c4-4ebc-bab7-543fe2991097","Type":"ContainerStarted","Data":"96f3a17077a296566b927cf1b8a4b0654601d3ff16ceb7a8fb5118be0d0b9215"}
Jul 28 02:52:44.447099 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:44.447012    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 02:52:45.671937 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:45.671899    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.69:8643/healthz\": dial tcp 10.133.0.69:8643: connect: connection refused"
Jul 28 02:52:46.859935 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:46.859906    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:52:46.985377 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:46.985293    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl2kp\" (UniqueName: \"kubernetes.io/projected/12f94aca-3a07-4a07-8f3c-98b8b920563b-kube-api-access-dl2kp\") pod \"12f94aca-3a07-4a07-8f3c-98b8b920563b\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") "
Jul 28 02:52:46.985377 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:46.985337    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12f94aca-3a07-4a07-8f3c-98b8b920563b-kube-rbac-proxy-sar-config\") pod \"12f94aca-3a07-4a07-8f3c-98b8b920563b\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") "
Jul 28 02:52:46.985670 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:46.985381    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12f94aca-3a07-4a07-8f3c-98b8b920563b-kserve-provision-location\") pod \"12f94aca-3a07-4a07-8f3c-98b8b920563b\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") "
Jul 28 02:52:46.985670 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:46.985426    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12f94aca-3a07-4a07-8f3c-98b8b920563b-proxy-tls\") pod \"12f94aca-3a07-4a07-8f3c-98b8b920563b\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") "
Jul 28 02:52:46.985670 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:46.985448    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12f94aca-3a07-4a07-8f3c-98b8b920563b-cabundle-cert\") pod \"12f94aca-3a07-4a07-8f3c-98b8b920563b\" (UID: \"12f94aca-3a07-4a07-8f3c-98b8b920563b\") "
Jul 28 02:52:46.985833 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:46.985765    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f94aca-3a07-4a07-8f3c-98b8b920563b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12f94aca-3a07-4a07-8f3c-98b8b920563b" (UID: "12f94aca-3a07-4a07-8f3c-98b8b920563b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:52:46.985965 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:46.985931    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f94aca-3a07-4a07-8f3c-98b8b920563b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "12f94aca-3a07-4a07-8f3c-98b8b920563b" (UID: "12f94aca-3a07-4a07-8f3c-98b8b920563b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:52:46.986297 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:46.985981    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f94aca-3a07-4a07-8f3c-98b8b920563b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "12f94aca-3a07-4a07-8f3c-98b8b920563b" (UID: "12f94aca-3a07-4a07-8f3c-98b8b920563b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:52:46.987951 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:46.987923    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f94aca-3a07-4a07-8f3c-98b8b920563b-kube-api-access-dl2kp" (OuterVolumeSpecName: "kube-api-access-dl2kp") pod "12f94aca-3a07-4a07-8f3c-98b8b920563b" (UID: "12f94aca-3a07-4a07-8f3c-98b8b920563b"). InnerVolumeSpecName "kube-api-access-dl2kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:52:46.988099 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:46.987990    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12f94aca-3a07-4a07-8f3c-98b8b920563b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12f94aca-3a07-4a07-8f3c-98b8b920563b" (UID: "12f94aca-3a07-4a07-8f3c-98b8b920563b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:52:46.988612 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:46.988585    2555 generic.go:356] "Generic (PLEG): container finished" podID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerID="b111324b026d6c817cc8e3320cd8181d577928291402fa28bd11ff71bdd2f89e" exitCode=0
Jul 28 02:52:46.988695 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:46.988662    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" event={"ID":"12f94aca-3a07-4a07-8f3c-98b8b920563b","Type":"ContainerDied","Data":"b111324b026d6c817cc8e3320cd8181d577928291402fa28bd11ff71bdd2f89e"}
Jul 28 02:52:46.988741 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:46.988721    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp" event={"ID":"12f94aca-3a07-4a07-8f3c-98b8b920563b","Type":"ContainerDied","Data":"c1f5476909cdd98cb2fd95eb586c7a8be4879b5c43ee08fea413b6170827a70b"}
Jul 28 02:52:46.988797 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:46.988747    2555 scope.go:117] "RemoveContainer" containerID="de5c3f67a701bdd3761e8f3ac74f4d47433d0188086cca92a152e22634c39bef"
Jul 28 02:52:46.988797 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:46.988671    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"
Jul 28 02:52:47.003332 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:47.003316    2555 scope.go:117] "RemoveContainer" containerID="b111324b026d6c817cc8e3320cd8181d577928291402fa28bd11ff71bdd2f89e"
Jul 28 02:52:47.010514 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:47.010495    2555 scope.go:117] "RemoveContainer" containerID="9a13248c3a52680799b5f9fc68e2a73c96f0d80e9f706d23c71c9235e6e3291a"
Jul 28 02:52:47.017400 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:47.017382    2555 scope.go:117] "RemoveContainer" containerID="de5c3f67a701bdd3761e8f3ac74f4d47433d0188086cca92a152e22634c39bef"
Jul 28 02:52:47.017696 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:52:47.017675    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5c3f67a701bdd3761e8f3ac74f4d47433d0188086cca92a152e22634c39bef\": container with ID starting with de5c3f67a701bdd3761e8f3ac74f4d47433d0188086cca92a152e22634c39bef not found: ID does not exist" containerID="de5c3f67a701bdd3761e8f3ac74f4d47433d0188086cca92a152e22634c39bef"
Jul 28 02:52:47.017770 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:47.017703    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5c3f67a701bdd3761e8f3ac74f4d47433d0188086cca92a152e22634c39bef"} err="failed to get container status \"de5c3f67a701bdd3761e8f3ac74f4d47433d0188086cca92a152e22634c39bef\": rpc error: code = NotFound desc = could not find container \"de5c3f67a701bdd3761e8f3ac74f4d47433d0188086cca92a152e22634c39bef\": container with ID starting with de5c3f67a701bdd3761e8f3ac74f4d47433d0188086cca92a152e22634c39bef not found: ID does not exist"
Jul 28 02:52:47.017770 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:47.017722    2555 scope.go:117] "RemoveContainer" containerID="b111324b026d6c817cc8e3320cd8181d577928291402fa28bd11ff71bdd2f89e"
Jul 28 02:52:47.018000 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:52:47.017982    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b111324b026d6c817cc8e3320cd8181d577928291402fa28bd11ff71bdd2f89e\": container with ID starting with b111324b026d6c817cc8e3320cd8181d577928291402fa28bd11ff71bdd2f89e not found: ID does not exist" containerID="b111324b026d6c817cc8e3320cd8181d577928291402fa28bd11ff71bdd2f89e"
Jul 28 02:52:47.018048 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:47.018003    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b111324b026d6c817cc8e3320cd8181d577928291402fa28bd11ff71bdd2f89e"} err="failed to get container status \"b111324b026d6c817cc8e3320cd8181d577928291402fa28bd11ff71bdd2f89e\": rpc error: code = NotFound desc = could not find container \"b111324b026d6c817cc8e3320cd8181d577928291402fa28bd11ff71bdd2f89e\": container with ID starting with b111324b026d6c817cc8e3320cd8181d577928291402fa28bd11ff71bdd2f89e not found: ID does not exist"
Jul 28 02:52:47.018048 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:47.018017    2555 scope.go:117] "RemoveContainer" containerID="9a13248c3a52680799b5f9fc68e2a73c96f0d80e9f706d23c71c9235e6e3291a"
Jul 28 02:52:47.018269 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:52:47.018246    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a13248c3a52680799b5f9fc68e2a73c96f0d80e9f706d23c71c9235e6e3291a\": container with ID starting with 9a13248c3a52680799b5f9fc68e2a73c96f0d80e9f706d23c71c9235e6e3291a not found: ID does not exist" containerID="9a13248c3a52680799b5f9fc68e2a73c96f0d80e9f706d23c71c9235e6e3291a"
Jul 28 02:52:47.018337 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:47.018280    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a13248c3a52680799b5f9fc68e2a73c96f0d80e9f706d23c71c9235e6e3291a"} err="failed to get container status \"9a13248c3a52680799b5f9fc68e2a73c96f0d80e9f706d23c71c9235e6e3291a\": rpc error: code = NotFound desc = could not find container \"9a13248c3a52680799b5f9fc68e2a73c96f0d80e9f706d23c71c9235e6e3291a\": container with ID starting with 9a13248c3a52680799b5f9fc68e2a73c96f0d80e9f706d23c71c9235e6e3291a not found: ID does not exist"
Jul 28 02:52:47.018647 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:47.018624    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"]
Jul 28 02:52:47.031558 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:47.031511    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-2vcsp"]
Jul 28 02:52:47.086791 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:47.086766    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12f94aca-3a07-4a07-8f3c-98b8b920563b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:52:47.086886 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:47.086790    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12f94aca-3a07-4a07-8f3c-98b8b920563b-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:52:47.086886 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:47.086806    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12f94aca-3a07-4a07-8f3c-98b8b920563b-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:52:47.086886 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:47.086819    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12f94aca-3a07-4a07-8f3c-98b8b920563b-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:52:47.086886 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:47.086830    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl2kp\" (UniqueName: \"kubernetes.io/projected/12f94aca-3a07-4a07-8f3c-98b8b920563b-kube-api-access-dl2kp\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:52:47.451677 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:47.451635    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" path="/var/lib/kubelet/pods/12f94aca-3a07-4a07-8f3c-98b8b920563b/volumes"
Jul 28 02:52:48.997217 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:48.997187    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b_af7e67d6-60c4-4ebc-bab7-543fe2991097/storage-initializer/0.log"
Jul 28 02:52:48.997623 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:48.997224    2555 generic.go:356] "Generic (PLEG): container finished" podID="af7e67d6-60c4-4ebc-bab7-543fe2991097" containerID="5eb71a038174702fcdcee02a98102a4b72260a82c880e545d0c80ab1f1d6a6aa" exitCode=1
Jul 28 02:52:48.997623 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:48.997266    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b" event={"ID":"af7e67d6-60c4-4ebc-bab7-543fe2991097","Type":"ContainerDied","Data":"5eb71a038174702fcdcee02a98102a4b72260a82c880e545d0c80ab1f1d6a6aa"}
Jul 28 02:52:50.001943 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:50.001915    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b_af7e67d6-60c4-4ebc-bab7-543fe2991097/storage-initializer/0.log"
Jul 28 02:52:50.002319 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:50.001986    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b" event={"ID":"af7e67d6-60c4-4ebc-bab7-543fe2991097","Type":"ContainerStarted","Data":"68051cbd44deb188fba248610bd34ad97d6d6f0f6bf2f397f3de61382fe8ce5d"}
Jul 28 02:52:51.007282 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:51.007202    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b_af7e67d6-60c4-4ebc-bab7-543fe2991097/storage-initializer/1.log"
Jul 28 02:52:51.007693 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:51.007593    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b_af7e67d6-60c4-4ebc-bab7-543fe2991097/storage-initializer/0.log"
Jul 28 02:52:51.007693 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:51.007626    2555 generic.go:356] "Generic (PLEG): container finished" podID="af7e67d6-60c4-4ebc-bab7-543fe2991097" containerID="68051cbd44deb188fba248610bd34ad97d6d6f0f6bf2f397f3de61382fe8ce5d" exitCode=1
Jul 28 02:52:51.007693 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:51.007662    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b" event={"ID":"af7e67d6-60c4-4ebc-bab7-543fe2991097","Type":"ContainerDied","Data":"68051cbd44deb188fba248610bd34ad97d6d6f0f6bf2f397f3de61382fe8ce5d"}
Jul 28 02:52:51.007823 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:51.007727    2555 scope.go:117] "RemoveContainer" containerID="5eb71a038174702fcdcee02a98102a4b72260a82c880e545d0c80ab1f1d6a6aa"
Jul 28 02:52:51.008065 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:51.008046    2555 scope.go:117] "RemoveContainer" containerID="5eb71a038174702fcdcee02a98102a4b72260a82c880e545d0c80ab1f1d6a6aa"
Jul 28 02:52:51.017990 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:52:51.017961    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b_kserve-ci-e2e-test(af7e67d6-60c4-4ebc-bab7-543fe2991097)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b" podUID="af7e67d6-60c4-4ebc-bab7-543fe2991097"
Jul 28 02:52:51.018082 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:52:51.018036    2555 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b_kserve-ci-e2e-test_af7e67d6-60c4-4ebc-bab7-543fe2991097_0 in pod sandbox 96f3a17077a296566b927cf1b8a4b0654601d3ff16ceb7a8fb5118be0d0b9215 from index: no such id: '5eb71a038174702fcdcee02a98102a4b72260a82c880e545d0c80ab1f1d6a6aa'" containerID="5eb71a038174702fcdcee02a98102a4b72260a82c880e545d0c80ab1f1d6a6aa"
Jul 28 02:52:51.018142 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:51.018074    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb71a038174702fcdcee02a98102a4b72260a82c880e545d0c80ab1f1d6a6aa"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b_kserve-ci-e2e-test_af7e67d6-60c4-4ebc-bab7-543fe2991097_0 in pod sandbox 96f3a17077a296566b927cf1b8a4b0654601d3ff16ceb7a8fb5118be0d0b9215 from index: no such id: '5eb71a038174702fcdcee02a98102a4b72260a82c880e545d0c80ab1f1d6a6aa'"
Jul 28 02:52:52.012852 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:52.012821    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b_af7e67d6-60c4-4ebc-bab7-543fe2991097/storage-initializer/1.log"
Jul 28 02:52:53.013741 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:53.013706    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"]
Jul 28 02:52:53.138153 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:53.138130    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b_af7e67d6-60c4-4ebc-bab7-543fe2991097/storage-initializer/1.log"
Jul 28 02:52:53.141601 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:53.141583    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"
Jul 28 02:52:53.234653 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:53.234612    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af7e67d6-60c4-4ebc-bab7-543fe2991097-proxy-tls\") pod \"af7e67d6-60c4-4ebc-bab7-543fe2991097\" (UID: \"af7e67d6-60c4-4ebc-bab7-543fe2991097\") "
Jul 28 02:52:53.234850 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:53.234684    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5k8b\" (UniqueName: \"kubernetes.io/projected/af7e67d6-60c4-4ebc-bab7-543fe2991097-kube-api-access-x5k8b\") pod \"af7e67d6-60c4-4ebc-bab7-543fe2991097\" (UID: \"af7e67d6-60c4-4ebc-bab7-543fe2991097\") "
Jul 28 02:52:53.234850 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:53.234718    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af7e67d6-60c4-4ebc-bab7-543fe2991097-kserve-provision-location\") pod \"af7e67d6-60c4-4ebc-bab7-543fe2991097\" (UID: \"af7e67d6-60c4-4ebc-bab7-543fe2991097\") "
Jul 28 02:52:53.234850 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:53.234744    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af7e67d6-60c4-4ebc-bab7-543fe2991097-kube-rbac-proxy-sar-config\") pod \"af7e67d6-60c4-4ebc-bab7-543fe2991097\" (UID: \"af7e67d6-60c4-4ebc-bab7-543fe2991097\") "
Jul 28 02:52:53.235066 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:53.235016    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7e67d6-60c4-4ebc-bab7-543fe2991097-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "af7e67d6-60c4-4ebc-bab7-543fe2991097" (UID: "af7e67d6-60c4-4ebc-bab7-543fe2991097"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:52:53.235170 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:53.235152    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7e67d6-60c4-4ebc-bab7-543fe2991097-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "af7e67d6-60c4-4ebc-bab7-543fe2991097" (UID: "af7e67d6-60c4-4ebc-bab7-543fe2991097"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:52:53.237043 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:53.237019    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af7e67d6-60c4-4ebc-bab7-543fe2991097-kube-api-access-x5k8b" (OuterVolumeSpecName: "kube-api-access-x5k8b") pod "af7e67d6-60c4-4ebc-bab7-543fe2991097" (UID: "af7e67d6-60c4-4ebc-bab7-543fe2991097"). InnerVolumeSpecName "kube-api-access-x5k8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:52:53.237122 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:53.237024    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7e67d6-60c4-4ebc-bab7-543fe2991097-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "af7e67d6-60c4-4ebc-bab7-543fe2991097" (UID: "af7e67d6-60c4-4ebc-bab7-543fe2991097"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:52:53.335564 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:53.335435    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5k8b\" (UniqueName: \"kubernetes.io/projected/af7e67d6-60c4-4ebc-bab7-543fe2991097-kube-api-access-x5k8b\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:52:53.335564 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:53.335464    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af7e67d6-60c4-4ebc-bab7-543fe2991097-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:52:53.335564 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:53.335474    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af7e67d6-60c4-4ebc-bab7-543fe2991097-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:52:53.335564 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:53.335485    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af7e67d6-60c4-4ebc-bab7-543fe2991097-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:52:54.020815 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.020785    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b_af7e67d6-60c4-4ebc-bab7-543fe2991097/storage-initializer/1.log"
Jul 28 02:52:54.023260 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.020861    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b" event={"ID":"af7e67d6-60c4-4ebc-bab7-543fe2991097","Type":"ContainerDied","Data":"96f3a17077a296566b927cf1b8a4b0654601d3ff16ceb7a8fb5118be0d0b9215"}
Jul 28 02:52:54.023260 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.020889    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"
Jul 28 02:52:54.023260 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.020927    2555 scope.go:117] "RemoveContainer" containerID="68051cbd44deb188fba248610bd34ad97d6d6f0f6bf2f397f3de61382fe8ce5d"
Jul 28 02:52:54.061104 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.061068    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"]
Jul 28 02:52:54.068592 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.068564    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-jdp7b"]
Jul 28 02:52:54.110463 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.110428    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"]
Jul 28 02:52:54.110702 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.110689    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7e67d6-60c4-4ebc-bab7-543fe2991097" containerName="storage-initializer"
Jul 28 02:52:54.110749 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.110704    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7e67d6-60c4-4ebc-bab7-543fe2991097" containerName="storage-initializer"
Jul 28 02:52:54.110749 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.110717    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="storage-initializer"
Jul 28 02:52:54.110749 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.110722    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="storage-initializer"
Jul 28 02:52:54.110749 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.110731    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kube-rbac-proxy"
Jul 28 02:52:54.110749 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.110738    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kube-rbac-proxy"
Jul 28 02:52:54.110749 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.110749    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kserve-container"
Jul 28 02:52:54.110936 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.110754    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kserve-container"
Jul 28 02:52:54.110936 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.110803    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="af7e67d6-60c4-4ebc-bab7-543fe2991097" containerName="storage-initializer"
Jul 28 02:52:54.110936 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.110811    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kube-rbac-proxy"
Jul 28 02:52:54.110936 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.110817    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="12f94aca-3a07-4a07-8f3c-98b8b920563b" containerName="kserve-container"
Jul 28 02:52:54.110936 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.110860    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7e67d6-60c4-4ebc-bab7-543fe2991097" containerName="storage-initializer"
Jul 28 02:52:54.110936 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.110865    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7e67d6-60c4-4ebc-bab7-543fe2991097" containerName="storage-initializer"
Jul 28 02:52:54.110936 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.110901    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="af7e67d6-60c4-4ebc-bab7-543fe2991097" containerName="storage-initializer"
Jul 28 02:52:54.114841 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.114825    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.117427 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.117388    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 02:52:54.117581 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.117433    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jul 28 02:52:54.117581 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.117468    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 02:52:54.118114 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.118095    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-96ksg\""
Jul 28 02:52:54.118339 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.118319    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jul 28 02:52:54.118430 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.118326    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:52:54.118430 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.118327    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 02:52:54.129041 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.129012    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"]
Jul 28 02:52:54.243705 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.243661    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cb32c67-0eda-418b-be6c-55072f6c04a3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.243705 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.243709    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxchg\" (UniqueName: \"kubernetes.io/projected/3cb32c67-0eda-418b-be6c-55072f6c04a3-kube-api-access-dxchg\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.243916 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.243789    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cb32c67-0eda-418b-be6c-55072f6c04a3-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.243916 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.243849    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cb32c67-0eda-418b-be6c-55072f6c04a3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.243916 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.243875    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cb32c67-0eda-418b-be6c-55072f6c04a3-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.344697 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.344615    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cb32c67-0eda-418b-be6c-55072f6c04a3-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.344697 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.344665    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cb32c67-0eda-418b-be6c-55072f6c04a3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.344697 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.344691    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cb32c67-0eda-418b-be6c-55072f6c04a3-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.344919 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.344721    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cb32c67-0eda-418b-be6c-55072f6c04a3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.344919 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.344748    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxchg\" (UniqueName: \"kubernetes.io/projected/3cb32c67-0eda-418b-be6c-55072f6c04a3-kube-api-access-dxchg\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.345239 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.345213    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cb32c67-0eda-418b-be6c-55072f6c04a3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.345427 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.345405    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cb32c67-0eda-418b-be6c-55072f6c04a3-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.345490 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.345407    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cb32c67-0eda-418b-be6c-55072f6c04a3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.347438 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.347419    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cb32c67-0eda-418b-be6c-55072f6c04a3-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.353460 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.353435    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxchg\" (UniqueName: \"kubernetes.io/projected/3cb32c67-0eda-418b-be6c-55072f6c04a3-kube-api-access-dxchg\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.430302 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.430262    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:54.586654 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:54.586614    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"]
Jul 28 02:52:54.589950 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:52:54.589925    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb32c67_0eda_418b_be6c_55072f6c04a3.slice/crio-01aa39b323ae52cf6800c10b6b6dc46b90f4c29fdccb3ab66e2a9de7745755cc WatchSource:0}: Error finding container 01aa39b323ae52cf6800c10b6b6dc46b90f4c29fdccb3ab66e2a9de7745755cc: Status 404 returned error can't find the container with id 01aa39b323ae52cf6800c10b6b6dc46b90f4c29fdccb3ab66e2a9de7745755cc
Jul 28 02:52:55.026096 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:55.026054    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" event={"ID":"3cb32c67-0eda-418b-be6c-55072f6c04a3","Type":"ContainerStarted","Data":"5d7836496722f39d7128ad60a88580f5747fafb61681a99f7a1435cebdb9f271"}
Jul 28 02:52:55.026539 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:55.026122    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" event={"ID":"3cb32c67-0eda-418b-be6c-55072f6c04a3","Type":"ContainerStarted","Data":"01aa39b323ae52cf6800c10b6b6dc46b90f4c29fdccb3ab66e2a9de7745755cc"}
Jul 28 02:52:55.452094 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:55.452060    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af7e67d6-60c4-4ebc-bab7-543fe2991097" path="/var/lib/kubelet/pods/af7e67d6-60c4-4ebc-bab7-543fe2991097/volumes"
Jul 28 02:52:56.032077 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:56.032044    2555 generic.go:356] "Generic (PLEG): container finished" podID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerID="5d7836496722f39d7128ad60a88580f5747fafb61681a99f7a1435cebdb9f271" exitCode=0
Jul 28 02:52:56.032479 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:56.032140    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" event={"ID":"3cb32c67-0eda-418b-be6c-55072f6c04a3","Type":"ContainerDied","Data":"5d7836496722f39d7128ad60a88580f5747fafb61681a99f7a1435cebdb9f271"}
Jul 28 02:52:57.037847 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:57.037807    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" event={"ID":"3cb32c67-0eda-418b-be6c-55072f6c04a3","Type":"ContainerStarted","Data":"3b7aee5dc54eb523fbc7cf057ed476da5b8e13c0903885c8e5c6ab207bc9a9b7"}
Jul 28 02:52:57.038242 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:57.037869    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" event={"ID":"3cb32c67-0eda-418b-be6c-55072f6c04a3","Type":"ContainerStarted","Data":"f6691b03c76cabc3152072e74a2719f2f2d6f90cf5a4d5c0c25789ed8ed8b65d"}
Jul 28 02:52:57.038242 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:57.037971    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:57.057454 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:57.057397    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" podStartSLOduration=2.088728568 podStartE2EDuration="3.057380654s" totalImagesPullingTime="0s" totalInitContainerRuntime="968.652086ms" isStatefulPod=false podCreationTimestamp="2026-07-28 02:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:52:57.056380529 +0000 UTC m=+5224.065898967" watchObservedRunningTime="2026-07-28 02:52:57.057380654 +0000 UTC m=+5224.066899061"
Jul 28 02:52:58.041210 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:58.041181    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:52:58.042322 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:58.042295    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 28 02:52:59.044709 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:52:59.044667    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 28 02:53:04.049290 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:53:04.049253    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:53:04.049745 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:53:04.049717    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 28 02:53:14.050352 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:53:14.050307    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 28 02:53:24.049770 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:53:24.049716    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 28 02:53:34.049771 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:53:34.049724    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 28 02:53:44.050458 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:53:44.050411    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 28 02:53:54.050539 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:53:54.050484    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 28 02:54:04.050677 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:04.050625    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 28 02:54:14.050746 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:14.050707    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:54:24.191795 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:24.191762    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"]
Jul 28 02:54:24.192174 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:24.191975    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kserve-container" containerID="cri-o://f6691b03c76cabc3152072e74a2719f2f2d6f90cf5a4d5c0c25789ed8ed8b65d" gracePeriod=30
Jul 28 02:54:24.192174 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:24.192079    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kube-rbac-proxy" containerID="cri-o://3b7aee5dc54eb523fbc7cf057ed476da5b8e13c0903885c8e5c6ab207bc9a9b7" gracePeriod=30
Jul 28 02:54:24.343956 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:24.343886    2555 generic.go:356] "Generic (PLEG): container finished" podID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerID="3b7aee5dc54eb523fbc7cf057ed476da5b8e13c0903885c8e5c6ab207bc9a9b7" exitCode=2
Jul 28 02:54:24.344075 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:24.343963    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" event={"ID":"3cb32c67-0eda-418b-be6c-55072f6c04a3","Type":"ContainerDied","Data":"3b7aee5dc54eb523fbc7cf057ed476da5b8e13c0903885c8e5c6ab207bc9a9b7"}
Jul 28 02:54:25.339028 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.338987    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"]
Jul 28 02:54:25.342170 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.342151    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"
Jul 28 02:54:25.344858 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.344835    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jul 28 02:54:25.344985 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.344869    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jul 28 02:54:25.353927 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.353896    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"]
Jul 28 02:54:25.411778 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.411742    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1650af0-cd93-4508-93b3-16f11f209d3f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2\" (UID: \"e1650af0-cd93-4508-93b3-16f11f209d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"
Jul 28 02:54:25.411952 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.411784    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1650af0-cd93-4508-93b3-16f11f209d3f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2\" (UID: \"e1650af0-cd93-4508-93b3-16f11f209d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"
Jul 28 02:54:25.411952 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.411855    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6qpb\" (UniqueName: \"kubernetes.io/projected/e1650af0-cd93-4508-93b3-16f11f209d3f-kube-api-access-w6qpb\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2\" (UID: \"e1650af0-cd93-4508-93b3-16f11f209d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"
Jul 28 02:54:25.412046 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.411948    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1650af0-cd93-4508-93b3-16f11f209d3f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2\" (UID: \"e1650af0-cd93-4508-93b3-16f11f209d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"
Jul 28 02:54:25.513229 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.513197    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1650af0-cd93-4508-93b3-16f11f209d3f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2\" (UID: \"e1650af0-cd93-4508-93b3-16f11f209d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"
Jul 28 02:54:25.513409 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.513243    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1650af0-cd93-4508-93b3-16f11f209d3f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2\" (UID: \"e1650af0-cd93-4508-93b3-16f11f209d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"
Jul 28 02:54:25.513409 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.513263    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1650af0-cd93-4508-93b3-16f11f209d3f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2\" (UID: \"e1650af0-cd93-4508-93b3-16f11f209d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"
Jul 28 02:54:25.513409 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.513286    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qpb\" (UniqueName: \"kubernetes.io/projected/e1650af0-cd93-4508-93b3-16f11f209d3f-kube-api-access-w6qpb\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2\" (UID: \"e1650af0-cd93-4508-93b3-16f11f209d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"
Jul 28 02:54:25.513705 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.513681    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1650af0-cd93-4508-93b3-16f11f209d3f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2\" (UID: \"e1650af0-cd93-4508-93b3-16f11f209d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"
Jul 28 02:54:25.513949 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.513927    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1650af0-cd93-4508-93b3-16f11f209d3f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2\" (UID: \"e1650af0-cd93-4508-93b3-16f11f209d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"
Jul 28 02:54:25.515960 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.515926    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1650af0-cd93-4508-93b3-16f11f209d3f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2\" (UID: \"e1650af0-cd93-4508-93b3-16f11f209d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"
Jul 28 02:54:25.522617 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.522591    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6qpb\" (UniqueName: \"kubernetes.io/projected/e1650af0-cd93-4508-93b3-16f11f209d3f-kube-api-access-w6qpb\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2\" (UID: \"e1650af0-cd93-4508-93b3-16f11f209d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"
Jul 28 02:54:25.652725 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.652631    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"
Jul 28 02:54:25.778835 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:25.778810    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"]
Jul 28 02:54:25.780877 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:54:25.780843    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1650af0_cd93_4508_93b3_16f11f209d3f.slice/crio-f6518219e61a2da0cfb66488efba031ac3e475b50b02e55cdaf0208e7ad9f61c WatchSource:0}: Error finding container f6518219e61a2da0cfb66488efba031ac3e475b50b02e55cdaf0208e7ad9f61c: Status 404 returned error can't find the container with id f6518219e61a2da0cfb66488efba031ac3e475b50b02e55cdaf0208e7ad9f61c
Jul 28 02:54:26.351865 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:26.351830    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2" event={"ID":"e1650af0-cd93-4508-93b3-16f11f209d3f","Type":"ContainerStarted","Data":"639df6397a99710a9af56c58d25c8772f47dc438f153208651d28fec6863629c"}
Jul 28 02:54:26.352240 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:26.351886    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2" event={"ID":"e1650af0-cd93-4508-93b3-16f11f209d3f","Type":"ContainerStarted","Data":"f6518219e61a2da0cfb66488efba031ac3e475b50b02e55cdaf0208e7ad9f61c"}
Jul 28 02:54:29.041301 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.041279    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:54:29.142384 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.142299    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxchg\" (UniqueName: \"kubernetes.io/projected/3cb32c67-0eda-418b-be6c-55072f6c04a3-kube-api-access-dxchg\") pod \"3cb32c67-0eda-418b-be6c-55072f6c04a3\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") "
Jul 28 02:54:29.142384 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.142358    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cb32c67-0eda-418b-be6c-55072f6c04a3-proxy-tls\") pod \"3cb32c67-0eda-418b-be6c-55072f6c04a3\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") "
Jul 28 02:54:29.142384 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.142387    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cb32c67-0eda-418b-be6c-55072f6c04a3-cabundle-cert\") pod \"3cb32c67-0eda-418b-be6c-55072f6c04a3\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") "
Jul 28 02:54:29.142698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.142503    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cb32c67-0eda-418b-be6c-55072f6c04a3-kube-rbac-proxy-sar-config\") pod \"3cb32c67-0eda-418b-be6c-55072f6c04a3\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") "
Jul 28 02:54:29.142698 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.142575    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cb32c67-0eda-418b-be6c-55072f6c04a3-kserve-provision-location\") pod \"3cb32c67-0eda-418b-be6c-55072f6c04a3\" (UID: \"3cb32c67-0eda-418b-be6c-55072f6c04a3\") "
Jul 28 02:54:29.142882 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.142777    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb32c67-0eda-418b-be6c-55072f6c04a3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3cb32c67-0eda-418b-be6c-55072f6c04a3" (UID: "3cb32c67-0eda-418b-be6c-55072f6c04a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:54:29.142939 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.142881    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb32c67-0eda-418b-be6c-55072f6c04a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3cb32c67-0eda-418b-be6c-55072f6c04a3" (UID: "3cb32c67-0eda-418b-be6c-55072f6c04a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:54:29.142939 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.142876    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb32c67-0eda-418b-be6c-55072f6c04a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3cb32c67-0eda-418b-be6c-55072f6c04a3" (UID: "3cb32c67-0eda-418b-be6c-55072f6c04a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:54:29.144663 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.144637    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb32c67-0eda-418b-be6c-55072f6c04a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3cb32c67-0eda-418b-be6c-55072f6c04a3" (UID: "3cb32c67-0eda-418b-be6c-55072f6c04a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:54:29.144759 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.144676    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb32c67-0eda-418b-be6c-55072f6c04a3-kube-api-access-dxchg" (OuterVolumeSpecName: "kube-api-access-dxchg") pod "3cb32c67-0eda-418b-be6c-55072f6c04a3" (UID: "3cb32c67-0eda-418b-be6c-55072f6c04a3"). InnerVolumeSpecName "kube-api-access-dxchg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:54:29.243188 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.243151    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cb32c67-0eda-418b-be6c-55072f6c04a3-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:54:29.243188 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.243179    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cb32c67-0eda-418b-be6c-55072f6c04a3-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:54:29.243188 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.243191    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cb32c67-0eda-418b-be6c-55072f6c04a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:54:29.243188 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.243200    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cb32c67-0eda-418b-be6c-55072f6c04a3-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:54:29.243459 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.243209    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxchg\" (UniqueName: \"kubernetes.io/projected/3cb32c67-0eda-418b-be6c-55072f6c04a3-kube-api-access-dxchg\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:54:29.364576 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.364541    2555 generic.go:356] "Generic (PLEG): container finished" podID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerID="f6691b03c76cabc3152072e74a2719f2f2d6f90cf5a4d5c0c25789ed8ed8b65d" exitCode=0
Jul 28 02:54:29.364708 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.364594    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"
Jul 28 02:54:29.364708 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.364599    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" event={"ID":"3cb32c67-0eda-418b-be6c-55072f6c04a3","Type":"ContainerDied","Data":"f6691b03c76cabc3152072e74a2719f2f2d6f90cf5a4d5c0c25789ed8ed8b65d"}
Jul 28 02:54:29.364708 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.364656    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q" event={"ID":"3cb32c67-0eda-418b-be6c-55072f6c04a3","Type":"ContainerDied","Data":"01aa39b323ae52cf6800c10b6b6dc46b90f4c29fdccb3ab66e2a9de7745755cc"}
Jul 28 02:54:29.364708 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.364682    2555 scope.go:117] "RemoveContainer" containerID="3b7aee5dc54eb523fbc7cf057ed476da5b8e13c0903885c8e5c6ab207bc9a9b7"
Jul 28 02:54:29.373093 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.373077    2555 scope.go:117] "RemoveContainer" containerID="f6691b03c76cabc3152072e74a2719f2f2d6f90cf5a4d5c0c25789ed8ed8b65d"
Jul 28 02:54:29.380206 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.380190    2555 scope.go:117] "RemoveContainer" containerID="5d7836496722f39d7128ad60a88580f5747fafb61681a99f7a1435cebdb9f271"
Jul 28 02:54:29.386103 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.386079    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"]
Jul 28 02:54:29.387599 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.387576    2555 scope.go:117] "RemoveContainer" containerID="3b7aee5dc54eb523fbc7cf057ed476da5b8e13c0903885c8e5c6ab207bc9a9b7"
Jul 28 02:54:29.387856 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:54:29.387837    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b7aee5dc54eb523fbc7cf057ed476da5b8e13c0903885c8e5c6ab207bc9a9b7\": container with ID starting with 3b7aee5dc54eb523fbc7cf057ed476da5b8e13c0903885c8e5c6ab207bc9a9b7 not found: ID does not exist" containerID="3b7aee5dc54eb523fbc7cf057ed476da5b8e13c0903885c8e5c6ab207bc9a9b7"
Jul 28 02:54:29.387909 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.387865    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b7aee5dc54eb523fbc7cf057ed476da5b8e13c0903885c8e5c6ab207bc9a9b7"} err="failed to get container status \"3b7aee5dc54eb523fbc7cf057ed476da5b8e13c0903885c8e5c6ab207bc9a9b7\": rpc error: code = NotFound desc = could not find container \"3b7aee5dc54eb523fbc7cf057ed476da5b8e13c0903885c8e5c6ab207bc9a9b7\": container with ID starting with 3b7aee5dc54eb523fbc7cf057ed476da5b8e13c0903885c8e5c6ab207bc9a9b7 not found: ID does not exist"
Jul 28 02:54:29.387909 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.387882    2555 scope.go:117] "RemoveContainer" containerID="f6691b03c76cabc3152072e74a2719f2f2d6f90cf5a4d5c0c25789ed8ed8b65d"
Jul 28 02:54:29.388090 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:54:29.388074    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6691b03c76cabc3152072e74a2719f2f2d6f90cf5a4d5c0c25789ed8ed8b65d\": container with ID starting with f6691b03c76cabc3152072e74a2719f2f2d6f90cf5a4d5c0c25789ed8ed8b65d not found: ID does not exist" containerID="f6691b03c76cabc3152072e74a2719f2f2d6f90cf5a4d5c0c25789ed8ed8b65d"
Jul 28 02:54:29.388136 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.388094    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6691b03c76cabc3152072e74a2719f2f2d6f90cf5a4d5c0c25789ed8ed8b65d"} err="failed to get container status \"f6691b03c76cabc3152072e74a2719f2f2d6f90cf5a4d5c0c25789ed8ed8b65d\": rpc error: code = NotFound desc = could not find container \"f6691b03c76cabc3152072e74a2719f2f2d6f90cf5a4d5c0c25789ed8ed8b65d\": container with ID starting with f6691b03c76cabc3152072e74a2719f2f2d6f90cf5a4d5c0c25789ed8ed8b65d not found: ID does not exist"
Jul 28 02:54:29.388136 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.388107    2555 scope.go:117] "RemoveContainer" containerID="5d7836496722f39d7128ad60a88580f5747fafb61681a99f7a1435cebdb9f271"
Jul 28 02:54:29.388346 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:54:29.388329    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7836496722f39d7128ad60a88580f5747fafb61681a99f7a1435cebdb9f271\": container with ID starting with 5d7836496722f39d7128ad60a88580f5747fafb61681a99f7a1435cebdb9f271 not found: ID does not exist" containerID="5d7836496722f39d7128ad60a88580f5747fafb61681a99f7a1435cebdb9f271"
Jul 28 02:54:29.388394 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.388351    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7836496722f39d7128ad60a88580f5747fafb61681a99f7a1435cebdb9f271"} err="failed to get container status \"5d7836496722f39d7128ad60a88580f5747fafb61681a99f7a1435cebdb9f271\": rpc error: code = NotFound desc = could not find container \"5d7836496722f39d7128ad60a88580f5747fafb61681a99f7a1435cebdb9f271\": container with ID starting with 5d7836496722f39d7128ad60a88580f5747fafb61681a99f7a1435cebdb9f271 not found: ID does not exist"
Jul 28 02:54:29.391836 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.391816    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-6j75q"]
Jul 28 02:54:29.451278 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:29.451198    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" path="/var/lib/kubelet/pods/3cb32c67-0eda-418b-be6c-55072f6c04a3/volumes"
Jul 28 02:54:31.373844 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:31.373814    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2_e1650af0-cd93-4508-93b3-16f11f209d3f/storage-initializer/0.log"
Jul 28 02:54:31.374203 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:31.373853    2555 generic.go:356] "Generic (PLEG): container finished" podID="e1650af0-cd93-4508-93b3-16f11f209d3f" containerID="639df6397a99710a9af56c58d25c8772f47dc438f153208651d28fec6863629c" exitCode=1
Jul 28 02:54:31.374203 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:31.373905    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2" event={"ID":"e1650af0-cd93-4508-93b3-16f11f209d3f","Type":"ContainerDied","Data":"639df6397a99710a9af56c58d25c8772f47dc438f153208651d28fec6863629c"}
Jul 28 02:54:32.378725 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:32.378698    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2_e1650af0-cd93-4508-93b3-16f11f209d3f/storage-initializer/0.log"
Jul 28 02:54:32.379148 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:32.378751    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2" event={"ID":"e1650af0-cd93-4508-93b3-16f11f209d3f","Type":"ContainerStarted","Data":"a06b7285a87d90a394f354aad733ca5259bdd3e77aa4dab35c450f6bec6a7997"}
Jul 28 02:54:35.246128 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:35.246091    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"]
Jul 28 02:54:35.246613 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:35.246334    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2" podUID="e1650af0-cd93-4508-93b3-16f11f209d3f" containerName="storage-initializer" containerID="cri-o://a06b7285a87d90a394f354aad733ca5259bdd3e77aa4dab35c450f6bec6a7997" gracePeriod=30
Jul 28 02:54:35.884280 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:35.884257    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2_e1650af0-cd93-4508-93b3-16f11f209d3f/storage-initializer/1.log"
Jul 28 02:54:35.884685 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:35.884670    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2_e1650af0-cd93-4508-93b3-16f11f209d3f/storage-initializer/0.log"
Jul 28 02:54:35.887068 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:35.887053    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"
Jul 28 02:54:35.993783 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:35.993758    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6qpb\" (UniqueName: \"kubernetes.io/projected/e1650af0-cd93-4508-93b3-16f11f209d3f-kube-api-access-w6qpb\") pod \"e1650af0-cd93-4508-93b3-16f11f209d3f\" (UID: \"e1650af0-cd93-4508-93b3-16f11f209d3f\") "
Jul 28 02:54:35.993919 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:35.993796    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1650af0-cd93-4508-93b3-16f11f209d3f-proxy-tls\") pod \"e1650af0-cd93-4508-93b3-16f11f209d3f\" (UID: \"e1650af0-cd93-4508-93b3-16f11f209d3f\") "
Jul 28 02:54:35.993919 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:35.993821    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1650af0-cd93-4508-93b3-16f11f209d3f-kserve-provision-location\") pod \"e1650af0-cd93-4508-93b3-16f11f209d3f\" (UID: \"e1650af0-cd93-4508-93b3-16f11f209d3f\") "
Jul 28 02:54:35.993919 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:35.993891    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1650af0-cd93-4508-93b3-16f11f209d3f-kube-rbac-proxy-sar-config\") pod \"e1650af0-cd93-4508-93b3-16f11f209d3f\" (UID: \"e1650af0-cd93-4508-93b3-16f11f209d3f\") "
Jul 28 02:54:35.994140 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:35.994120    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1650af0-cd93-4508-93b3-16f11f209d3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e1650af0-cd93-4508-93b3-16f11f209d3f" (UID: "e1650af0-cd93-4508-93b3-16f11f209d3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:54:35.994308 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:35.994278    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1650af0-cd93-4508-93b3-16f11f209d3f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e1650af0-cd93-4508-93b3-16f11f209d3f" (UID: "e1650af0-cd93-4508-93b3-16f11f209d3f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:54:35.995963 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:35.995944    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1650af0-cd93-4508-93b3-16f11f209d3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1650af0-cd93-4508-93b3-16f11f209d3f" (UID: "e1650af0-cd93-4508-93b3-16f11f209d3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:54:35.996047 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:35.995971    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1650af0-cd93-4508-93b3-16f11f209d3f-kube-api-access-w6qpb" (OuterVolumeSpecName: "kube-api-access-w6qpb") pod "e1650af0-cd93-4508-93b3-16f11f209d3f" (UID: "e1650af0-cd93-4508-93b3-16f11f209d3f"). InnerVolumeSpecName "kube-api-access-w6qpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:54:36.094473 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.094431    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1650af0-cd93-4508-93b3-16f11f209d3f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:54:36.094473 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.094470    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6qpb\" (UniqueName: \"kubernetes.io/projected/e1650af0-cd93-4508-93b3-16f11f209d3f-kube-api-access-w6qpb\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:54:36.094473 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.094480    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1650af0-cd93-4508-93b3-16f11f209d3f-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:54:36.094473 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.094488    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1650af0-cd93-4508-93b3-16f11f209d3f-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:54:36.387714 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.387637    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"]
Jul 28 02:54:36.388053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.387871    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="storage-initializer"
Jul 28 02:54:36.388053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.387881    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="storage-initializer"
Jul 28 02:54:36.388053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.387895    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kube-rbac-proxy"
Jul 28 02:54:36.388053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.387900    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kube-rbac-proxy"
Jul 28 02:54:36.388053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.387907    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1650af0-cd93-4508-93b3-16f11f209d3f" containerName="storage-initializer"
Jul 28 02:54:36.388053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.387912    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1650af0-cd93-4508-93b3-16f11f209d3f" containerName="storage-initializer"
Jul 28 02:54:36.388053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.387919    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kserve-container"
Jul 28 02:54:36.388053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.387924    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kserve-container"
Jul 28 02:54:36.388053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.387932    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1650af0-cd93-4508-93b3-16f11f209d3f" containerName="storage-initializer"
Jul 28 02:54:36.388053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.387937    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1650af0-cd93-4508-93b3-16f11f209d3f" containerName="storage-initializer"
Jul 28 02:54:36.388053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.387977    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1650af0-cd93-4508-93b3-16f11f209d3f" containerName="storage-initializer"
Jul 28 02:54:36.388053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.387984    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kserve-container"
Jul 28 02:54:36.388053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.387989    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb32c67-0eda-418b-be6c-55072f6c04a3" containerName="kube-rbac-proxy"
Jul 28 02:54:36.388053 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.388059    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1650af0-cd93-4508-93b3-16f11f209d3f" containerName="storage-initializer"
Jul 28 02:54:36.392251 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.391479    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.396844 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.394747    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:54:36.396844 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.394972    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jul 28 02:54:36.396844 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.395028    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jul 28 02:54:36.396844 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.395673    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2_e1650af0-cd93-4508-93b3-16f11f209d3f/storage-initializer/1.log"
Jul 28 02:54:36.397487 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.397467    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2_e1650af0-cd93-4508-93b3-16f11f209d3f/storage-initializer/0.log"
Jul 28 02:54:36.397606 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.397515    2555 generic.go:356] "Generic (PLEG): container finished" podID="e1650af0-cd93-4508-93b3-16f11f209d3f" containerID="a06b7285a87d90a394f354aad733ca5259bdd3e77aa4dab35c450f6bec6a7997" exitCode=1
Jul 28 02:54:36.397606 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.397561    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2" event={"ID":"e1650af0-cd93-4508-93b3-16f11f209d3f","Type":"ContainerDied","Data":"a06b7285a87d90a394f354aad733ca5259bdd3e77aa4dab35c450f6bec6a7997"}
Jul 28 02:54:36.397606 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.397589    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"
Jul 28 02:54:36.397760 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.397661    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2" event={"ID":"e1650af0-cd93-4508-93b3-16f11f209d3f","Type":"ContainerDied","Data":"f6518219e61a2da0cfb66488efba031ac3e475b50b02e55cdaf0208e7ad9f61c"}
Jul 28 02:54:36.397760 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.397691    2555 scope.go:117] "RemoveContainer" containerID="a06b7285a87d90a394f354aad733ca5259bdd3e77aa4dab35c450f6bec6a7997"
Jul 28 02:54:36.401615 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.401590    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"]
Jul 28 02:54:36.408171 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.408148    2555 scope.go:117] "RemoveContainer" containerID="639df6397a99710a9af56c58d25c8772f47dc438f153208651d28fec6863629c"
Jul 28 02:54:36.416026 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.416009    2555 scope.go:117] "RemoveContainer" containerID="a06b7285a87d90a394f354aad733ca5259bdd3e77aa4dab35c450f6bec6a7997"
Jul 28 02:54:36.416271 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:54:36.416251    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06b7285a87d90a394f354aad733ca5259bdd3e77aa4dab35c450f6bec6a7997\": container with ID starting with a06b7285a87d90a394f354aad733ca5259bdd3e77aa4dab35c450f6bec6a7997 not found: ID does not exist" containerID="a06b7285a87d90a394f354aad733ca5259bdd3e77aa4dab35c450f6bec6a7997"
Jul 28 02:54:36.416346 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.416283    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06b7285a87d90a394f354aad733ca5259bdd3e77aa4dab35c450f6bec6a7997"} err="failed to get container status \"a06b7285a87d90a394f354aad733ca5259bdd3e77aa4dab35c450f6bec6a7997\": rpc error: code = NotFound desc = could not find container \"a06b7285a87d90a394f354aad733ca5259bdd3e77aa4dab35c450f6bec6a7997\": container with ID starting with a06b7285a87d90a394f354aad733ca5259bdd3e77aa4dab35c450f6bec6a7997 not found: ID does not exist"
Jul 28 02:54:36.416346 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.416306    2555 scope.go:117] "RemoveContainer" containerID="639df6397a99710a9af56c58d25c8772f47dc438f153208651d28fec6863629c"
Jul 28 02:54:36.416633 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:54:36.416609    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639df6397a99710a9af56c58d25c8772f47dc438f153208651d28fec6863629c\": container with ID starting with 639df6397a99710a9af56c58d25c8772f47dc438f153208651d28fec6863629c not found: ID does not exist" containerID="639df6397a99710a9af56c58d25c8772f47dc438f153208651d28fec6863629c"
Jul 28 02:54:36.416700 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.416643    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639df6397a99710a9af56c58d25c8772f47dc438f153208651d28fec6863629c"} err="failed to get container status \"639df6397a99710a9af56c58d25c8772f47dc438f153208651d28fec6863629c\": rpc error: code = NotFound desc = could not find container \"639df6397a99710a9af56c58d25c8772f47dc438f153208651d28fec6863629c\": container with ID starting with 639df6397a99710a9af56c58d25c8772f47dc438f153208651d28fec6863629c not found: ID does not exist"
Jul 28 02:54:36.452943 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.452911    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"]
Jul 28 02:54:36.458033 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.458007    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-bfsp2"]
Jul 28 02:54:36.497587 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.497558    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/726e7cda-079e-4c81-bb0f-68fc2cba10e7-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.497733 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.497590    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/726e7cda-079e-4c81-bb0f-68fc2cba10e7-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.497733 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.497667    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.497733 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.497711    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.497878 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.497752    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk7wg\" (UniqueName: \"kubernetes.io/projected/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kube-api-access-qk7wg\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.598338 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.598304    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.598338 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.598337    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.598582 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.598362    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk7wg\" (UniqueName: \"kubernetes.io/projected/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kube-api-access-qk7wg\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.598582 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.598378    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/726e7cda-079e-4c81-bb0f-68fc2cba10e7-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.598582 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.598395    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/726e7cda-079e-4c81-bb0f-68fc2cba10e7-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.598871 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.598847    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.599071 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.599048    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.599119 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.599078    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/726e7cda-079e-4c81-bb0f-68fc2cba10e7-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.601009 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.600992    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/726e7cda-079e-4c81-bb0f-68fc2cba10e7-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.607189 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.607171    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk7wg\" (UniqueName: \"kubernetes.io/projected/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kube-api-access-qk7wg\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.709364 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.709298    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:36.840336 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:36.840306    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"]
Jul 28 02:54:36.843783 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:54:36.843746    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726e7cda_079e_4c81_bb0f_68fc2cba10e7.slice/crio-0abbde5763b216c67968971bd08b710bae0ace677af3c4c42b45adc1d8f323f9 WatchSource:0}: Error finding container 0abbde5763b216c67968971bd08b710bae0ace677af3c4c42b45adc1d8f323f9: Status 404 returned error can't find the container with id 0abbde5763b216c67968971bd08b710bae0ace677af3c4c42b45adc1d8f323f9
Jul 28 02:54:37.403743 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:37.403709    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" event={"ID":"726e7cda-079e-4c81-bb0f-68fc2cba10e7","Type":"ContainerStarted","Data":"4a7f37d786a16dbe297427a797edd0356b8bbf4dc9fc0300dcc29ed6ab8ebdbe"}
Jul 28 02:54:37.404176 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:37.403772    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" event={"ID":"726e7cda-079e-4c81-bb0f-68fc2cba10e7","Type":"ContainerStarted","Data":"0abbde5763b216c67968971bd08b710bae0ace677af3c4c42b45adc1d8f323f9"}
Jul 28 02:54:37.452165 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:37.452129    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1650af0-cd93-4508-93b3-16f11f209d3f" path="/var/lib/kubelet/pods/e1650af0-cd93-4508-93b3-16f11f209d3f/volumes"
Jul 28 02:54:38.408505 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:38.408466    2555 generic.go:356] "Generic (PLEG): container finished" podID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerID="4a7f37d786a16dbe297427a797edd0356b8bbf4dc9fc0300dcc29ed6ab8ebdbe" exitCode=0
Jul 28 02:54:38.408896 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:38.408562    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" event={"ID":"726e7cda-079e-4c81-bb0f-68fc2cba10e7","Type":"ContainerDied","Data":"4a7f37d786a16dbe297427a797edd0356b8bbf4dc9fc0300dcc29ed6ab8ebdbe"}
Jul 28 02:54:39.413851 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:39.413818    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" event={"ID":"726e7cda-079e-4c81-bb0f-68fc2cba10e7","Type":"ContainerStarted","Data":"47c1b7a4ea0391bbe2a3523f54a3bc806762472f705d7ee6e7d7d1a384795a8f"}
Jul 28 02:54:39.414281 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:39.413874    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" event={"ID":"726e7cda-079e-4c81-bb0f-68fc2cba10e7","Type":"ContainerStarted","Data":"8583a1ef617192606f4e551f31c3e2c1c7d19eb86c3de708e578680e9498dbcf"}
Jul 28 02:54:39.414281 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:39.413946    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:39.434858 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:39.434810    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" podStartSLOduration=2.525025701 podStartE2EDuration="3.434795097s" totalImagesPullingTime="0s" totalInitContainerRuntime="909.769396ms" isStatefulPod=false podCreationTimestamp="2026-07-28 02:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:54:39.433050025 +0000 UTC m=+5326.442568948" watchObservedRunningTime="2026-07-28 02:54:39.434795097 +0000 UTC m=+5326.444313521"
Jul 28 02:54:40.417808 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:40.417773    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:40.419034 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:40.419005    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.73:8080: connect: connection refused"
Jul 28 02:54:41.421186 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:41.421142    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.73:8080: connect: connection refused"
Jul 28 02:54:46.426295 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:46.426259    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:54:46.426755 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:46.426730    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.73:8080: connect: connection refused"
Jul 28 02:54:56.427579 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:54:56.427516    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.73:8080: connect: connection refused"
Jul 28 02:55:06.426852 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:55:06.426812    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.73:8080: connect: connection refused"
Jul 28 02:55:16.427101 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:55:16.427061    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.73:8080: connect: connection refused"
Jul 28 02:55:26.427355 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:55:26.427314    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.73:8080: connect: connection refused"
Jul 28 02:55:36.427255 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:55:36.427212    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.73:8080: connect: connection refused"
Jul 28 02:55:46.427410 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:55:46.427327    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.73:8080: connect: connection refused"
Jul 28 02:55:56.427490 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:55:56.427453    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:56:06.454808 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:06.454774    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"]
Jul 28 02:56:06.455228 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:06.455016    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kserve-container" containerID="cri-o://8583a1ef617192606f4e551f31c3e2c1c7d19eb86c3de708e578680e9498dbcf" gracePeriod=30
Jul 28 02:56:06.455228 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:06.455050    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kube-rbac-proxy" containerID="cri-o://47c1b7a4ea0391bbe2a3523f54a3bc806762472f705d7ee6e7d7d1a384795a8f" gracePeriod=30
Jul 28 02:56:06.710692 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:06.710609    2555 generic.go:356] "Generic (PLEG): container finished" podID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerID="47c1b7a4ea0391bbe2a3523f54a3bc806762472f705d7ee6e7d7d1a384795a8f" exitCode=2
Jul 28 02:56:06.710826 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:06.710679    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" event={"ID":"726e7cda-079e-4c81-bb0f-68fc2cba10e7","Type":"ContainerDied","Data":"47c1b7a4ea0391bbe2a3523f54a3bc806762472f705d7ee6e7d7d1a384795a8f"}
Jul 28 02:56:07.550173 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.550133    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"]
Jul 28 02:56:07.553454 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.553434    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"
Jul 28 02:56:07.556076 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.556054    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jul 28 02:56:07.556182 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.556078    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jul 28 02:56:07.563600 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.563575    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"]
Jul 28 02:56:07.602935 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.602899    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv\" (UID: \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"
Jul 28 02:56:07.603059 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.602941    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbz6h\" (UniqueName: \"kubernetes.io/projected/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kube-api-access-kbz6h\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv\" (UID: \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"
Jul 28 02:56:07.603059 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.602972    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv\" (UID: \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"
Jul 28 02:56:07.603059 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.603030    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv\" (UID: \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"
Jul 28 02:56:07.703580 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.703542    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv\" (UID: \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"
Jul 28 02:56:07.703580 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.703582    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbz6h\" (UniqueName: \"kubernetes.io/projected/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kube-api-access-kbz6h\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv\" (UID: \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"
Jul 28 02:56:07.703845 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.703607    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv\" (UID: \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"
Jul 28 02:56:07.703845 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.703789    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv\" (UID: \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"
Jul 28 02:56:07.703954 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.703944    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv\" (UID: \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"
Jul 28 02:56:07.704220 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.704196    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv\" (UID: \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"
Jul 28 02:56:07.706211 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.706192    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv\" (UID: \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"
Jul 28 02:56:07.712403 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.712374    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbz6h\" (UniqueName: \"kubernetes.io/projected/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kube-api-access-kbz6h\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv\" (UID: \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"
Jul 28 02:56:07.865071 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.864987    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"
Jul 28 02:56:07.995912 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:07.995877    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"]
Jul 28 02:56:07.999433 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:56:07.999405    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de9f5b2_0547_4584_84c0_7de4b57bd2dc.slice/crio-d3c1ecf4a2dc5f952c485142ed866ece8ef9a57bc2ecd378db24f096d4533a36 WatchSource:0}: Error finding container d3c1ecf4a2dc5f952c485142ed866ece8ef9a57bc2ecd378db24f096d4533a36: Status 404 returned error can't find the container with id d3c1ecf4a2dc5f952c485142ed866ece8ef9a57bc2ecd378db24f096d4533a36
Jul 28 02:56:08.721856 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:08.721817    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv" event={"ID":"9de9f5b2-0547-4584-84c0-7de4b57bd2dc","Type":"ContainerStarted","Data":"02ccb82870ad0c2a8ba48160766d135795f45326416cbc58eaaf0a1425b319eb"}
Jul 28 02:56:08.722210 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:08.721877    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv" event={"ID":"9de9f5b2-0547-4584-84c0-7de4b57bd2dc","Type":"ContainerStarted","Data":"d3c1ecf4a2dc5f952c485142ed866ece8ef9a57bc2ecd378db24f096d4533a36"}
Jul 28 02:56:11.202916 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.202890    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:56:11.226587 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.226473    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kube-rbac-proxy-sar-config\") pod \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") "
Jul 28 02:56:11.226587 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.226545    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kserve-provision-location\") pod \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") "
Jul 28 02:56:11.226827 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.226600    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/726e7cda-079e-4c81-bb0f-68fc2cba10e7-proxy-tls\") pod \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") "
Jul 28 02:56:11.226827 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.226641    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk7wg\" (UniqueName: \"kubernetes.io/projected/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kube-api-access-qk7wg\") pod \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") "
Jul 28 02:56:11.226827 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.226709    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/726e7cda-079e-4c81-bb0f-68fc2cba10e7-cabundle-cert\") pod \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\" (UID: \"726e7cda-079e-4c81-bb0f-68fc2cba10e7\") "
Jul 28 02:56:11.227292 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.227175    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726e7cda-079e-4c81-bb0f-68fc2cba10e7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "726e7cda-079e-4c81-bb0f-68fc2cba10e7" (UID: "726e7cda-079e-4c81-bb0f-68fc2cba10e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:56:11.227399 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.227321    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "726e7cda-079e-4c81-bb0f-68fc2cba10e7" (UID: "726e7cda-079e-4c81-bb0f-68fc2cba10e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:56:11.232550 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.228882    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "726e7cda-079e-4c81-bb0f-68fc2cba10e7" (UID: "726e7cda-079e-4c81-bb0f-68fc2cba10e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:56:11.241029 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.240979    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kube-api-access-qk7wg" (OuterVolumeSpecName: "kube-api-access-qk7wg") pod "726e7cda-079e-4c81-bb0f-68fc2cba10e7" (UID: "726e7cda-079e-4c81-bb0f-68fc2cba10e7"). InnerVolumeSpecName "kube-api-access-qk7wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:56:11.242195 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.242166    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/726e7cda-079e-4c81-bb0f-68fc2cba10e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "726e7cda-079e-4c81-bb0f-68fc2cba10e7" (UID: "726e7cda-079e-4c81-bb0f-68fc2cba10e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:56:11.327839 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.327810    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:56:11.327839 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.327837    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:56:11.327997 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.327848    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/726e7cda-079e-4c81-bb0f-68fc2cba10e7-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:56:11.327997 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.327859    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk7wg\" (UniqueName: \"kubernetes.io/projected/726e7cda-079e-4c81-bb0f-68fc2cba10e7-kube-api-access-qk7wg\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:56:11.327997 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.327868    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/726e7cda-079e-4c81-bb0f-68fc2cba10e7-cabundle-cert\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:56:11.733101 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.733073    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv_9de9f5b2-0547-4584-84c0-7de4b57bd2dc/storage-initializer/0.log"
Jul 28 02:56:11.733273 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.733115    2555 generic.go:356] "Generic (PLEG): container finished" podID="9de9f5b2-0547-4584-84c0-7de4b57bd2dc" containerID="02ccb82870ad0c2a8ba48160766d135795f45326416cbc58eaaf0a1425b319eb" exitCode=1
Jul 28 02:56:11.733273 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.733193    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv" event={"ID":"9de9f5b2-0547-4584-84c0-7de4b57bd2dc","Type":"ContainerDied","Data":"02ccb82870ad0c2a8ba48160766d135795f45326416cbc58eaaf0a1425b319eb"}
Jul 28 02:56:11.735010 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.734988    2555 generic.go:356] "Generic (PLEG): container finished" podID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerID="8583a1ef617192606f4e551f31c3e2c1c7d19eb86c3de708e578680e9498dbcf" exitCode=0
Jul 28 02:56:11.735091 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.735022    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" event={"ID":"726e7cda-079e-4c81-bb0f-68fc2cba10e7","Type":"ContainerDied","Data":"8583a1ef617192606f4e551f31c3e2c1c7d19eb86c3de708e578680e9498dbcf"}
Jul 28 02:56:11.735091 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.735054    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"
Jul 28 02:56:11.735091 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.735064    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc" event={"ID":"726e7cda-079e-4c81-bb0f-68fc2cba10e7","Type":"ContainerDied","Data":"0abbde5763b216c67968971bd08b710bae0ace677af3c4c42b45adc1d8f323f9"}
Jul 28 02:56:11.735091 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.735090    2555 scope.go:117] "RemoveContainer" containerID="47c1b7a4ea0391bbe2a3523f54a3bc806762472f705d7ee6e7d7d1a384795a8f"
Jul 28 02:56:11.743290 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.743272    2555 scope.go:117] "RemoveContainer" containerID="8583a1ef617192606f4e551f31c3e2c1c7d19eb86c3de708e578680e9498dbcf"
Jul 28 02:56:11.750747 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.750723    2555 scope.go:117] "RemoveContainer" containerID="4a7f37d786a16dbe297427a797edd0356b8bbf4dc9fc0300dcc29ed6ab8ebdbe"
Jul 28 02:56:11.762215 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.762199    2555 scope.go:117] "RemoveContainer" containerID="47c1b7a4ea0391bbe2a3523f54a3bc806762472f705d7ee6e7d7d1a384795a8f"
Jul 28 02:56:11.762476 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:56:11.762456    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c1b7a4ea0391bbe2a3523f54a3bc806762472f705d7ee6e7d7d1a384795a8f\": container with ID starting with 47c1b7a4ea0391bbe2a3523f54a3bc806762472f705d7ee6e7d7d1a384795a8f not found: ID does not exist" containerID="47c1b7a4ea0391bbe2a3523f54a3bc806762472f705d7ee6e7d7d1a384795a8f"
Jul 28 02:56:11.762589 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.762490    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c1b7a4ea0391bbe2a3523f54a3bc806762472f705d7ee6e7d7d1a384795a8f"} err="failed to get container status \"47c1b7a4ea0391bbe2a3523f54a3bc806762472f705d7ee6e7d7d1a384795a8f\": rpc error: code = NotFound desc = could not find container \"47c1b7a4ea0391bbe2a3523f54a3bc806762472f705d7ee6e7d7d1a384795a8f\": container with ID starting with 47c1b7a4ea0391bbe2a3523f54a3bc806762472f705d7ee6e7d7d1a384795a8f not found: ID does not exist"
Jul 28 02:56:11.762589 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.762515    2555 scope.go:117] "RemoveContainer" containerID="8583a1ef617192606f4e551f31c3e2c1c7d19eb86c3de708e578680e9498dbcf"
Jul 28 02:56:11.762885 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:56:11.762860    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8583a1ef617192606f4e551f31c3e2c1c7d19eb86c3de708e578680e9498dbcf\": container with ID starting with 8583a1ef617192606f4e551f31c3e2c1c7d19eb86c3de708e578680e9498dbcf not found: ID does not exist" containerID="8583a1ef617192606f4e551f31c3e2c1c7d19eb86c3de708e578680e9498dbcf"
Jul 28 02:56:11.762943 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.762897    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8583a1ef617192606f4e551f31c3e2c1c7d19eb86c3de708e578680e9498dbcf"} err="failed to get container status \"8583a1ef617192606f4e551f31c3e2c1c7d19eb86c3de708e578680e9498dbcf\": rpc error: code = NotFound desc = could not find container \"8583a1ef617192606f4e551f31c3e2c1c7d19eb86c3de708e578680e9498dbcf\": container with ID starting with 8583a1ef617192606f4e551f31c3e2c1c7d19eb86c3de708e578680e9498dbcf not found: ID does not exist"
Jul 28 02:56:11.762943 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.762920    2555 scope.go:117] "RemoveContainer" containerID="4a7f37d786a16dbe297427a797edd0356b8bbf4dc9fc0300dcc29ed6ab8ebdbe"
Jul 28 02:56:11.763154 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:56:11.763137    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7f37d786a16dbe297427a797edd0356b8bbf4dc9fc0300dcc29ed6ab8ebdbe\": container with ID starting with 4a7f37d786a16dbe297427a797edd0356b8bbf4dc9fc0300dcc29ed6ab8ebdbe not found: ID does not exist" containerID="4a7f37d786a16dbe297427a797edd0356b8bbf4dc9fc0300dcc29ed6ab8ebdbe"
Jul 28 02:56:11.763208 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.763160    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7f37d786a16dbe297427a797edd0356b8bbf4dc9fc0300dcc29ed6ab8ebdbe"} err="failed to get container status \"4a7f37d786a16dbe297427a797edd0356b8bbf4dc9fc0300dcc29ed6ab8ebdbe\": rpc error: code = NotFound desc = could not find container \"4a7f37d786a16dbe297427a797edd0356b8bbf4dc9fc0300dcc29ed6ab8ebdbe\": container with ID starting with 4a7f37d786a16dbe297427a797edd0356b8bbf4dc9fc0300dcc29ed6ab8ebdbe not found: ID does not exist"
Jul 28 02:56:11.769228 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.769205    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"]
Jul 28 02:56:11.773270 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:11.773247    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-p7gjc"]
Jul 28 02:56:12.741281 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:12.741251    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv_9de9f5b2-0547-4584-84c0-7de4b57bd2dc/storage-initializer/0.log"
Jul 28 02:56:12.741741 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:12.741303    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv" event={"ID":"9de9f5b2-0547-4584-84c0-7de4b57bd2dc","Type":"ContainerStarted","Data":"4d7d5d3e252f91a682aa6263a9acc84bd6c87d7d9108df00f2e6cbfca4bb43ce"}
Jul 28 02:56:13.451538 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:13.451485    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" path="/var/lib/kubelet/pods/726e7cda-079e-4c81-bb0f-68fc2cba10e7/volumes"
Jul 28 02:56:14.749517 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:14.749490    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv_9de9f5b2-0547-4584-84c0-7de4b57bd2dc/storage-initializer/1.log"
Jul 28 02:56:14.749905 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:14.749890    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv_9de9f5b2-0547-4584-84c0-7de4b57bd2dc/storage-initializer/0.log"
Jul 28 02:56:14.749945 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:14.749923    2555 generic.go:356] "Generic (PLEG): container finished" podID="9de9f5b2-0547-4584-84c0-7de4b57bd2dc" containerID="4d7d5d3e252f91a682aa6263a9acc84bd6c87d7d9108df00f2e6cbfca4bb43ce" exitCode=1
Jul 28 02:56:14.749985 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:14.749976    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv" event={"ID":"9de9f5b2-0547-4584-84c0-7de4b57bd2dc","Type":"ContainerDied","Data":"4d7d5d3e252f91a682aa6263a9acc84bd6c87d7d9108df00f2e6cbfca4bb43ce"}
Jul 28 02:56:14.750038 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:14.750028    2555 scope.go:117] "RemoveContainer" containerID="02ccb82870ad0c2a8ba48160766d135795f45326416cbc58eaaf0a1425b319eb"
Jul 28 02:56:14.750363 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:14.750346    2555 scope.go:117] "RemoveContainer" containerID="02ccb82870ad0c2a8ba48160766d135795f45326416cbc58eaaf0a1425b319eb"
Jul 28 02:56:14.759855 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:56:14.759831    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv_kserve-ci-e2e-test(9de9f5b2-0547-4584-84c0-7de4b57bd2dc)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv" podUID="9de9f5b2-0547-4584-84c0-7de4b57bd2dc"
Jul 28 02:56:14.759923 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:56:14.759899    2555 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv_kserve-ci-e2e-test_9de9f5b2-0547-4584-84c0-7de4b57bd2dc_0 in pod sandbox d3c1ecf4a2dc5f952c485142ed866ece8ef9a57bc2ecd378db24f096d4533a36 from index: no such id: '02ccb82870ad0c2a8ba48160766d135795f45326416cbc58eaaf0a1425b319eb'" containerID="02ccb82870ad0c2a8ba48160766d135795f45326416cbc58eaaf0a1425b319eb"
Jul 28 02:56:14.759969 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:14.759931    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02ccb82870ad0c2a8ba48160766d135795f45326416cbc58eaaf0a1425b319eb"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv_kserve-ci-e2e-test_9de9f5b2-0547-4584-84c0-7de4b57bd2dc_0 in pod sandbox d3c1ecf4a2dc5f952c485142ed866ece8ef9a57bc2ecd378db24f096d4533a36 from index: no such id: '02ccb82870ad0c2a8ba48160766d135795f45326416cbc58eaaf0a1425b319eb'"
Jul 28 02:56:15.754833 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:15.754807    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv_9de9f5b2-0547-4584-84c0-7de4b57bd2dc/storage-initializer/1.log"
Jul 28 02:56:17.527661 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.527630    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"]
Jul 28 02:56:17.648131 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.648107    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv_9de9f5b2-0547-4584-84c0-7de4b57bd2dc/storage-initializer/1.log"
Jul 28 02:56:17.651662 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.651645    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"
Jul 28 02:56:17.672011 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.671978    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kserve-provision-location\") pod \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\" (UID: \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\") "
Jul 28 02:56:17.672160 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.672038    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kube-rbac-proxy-sar-config\") pod \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\" (UID: \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\") "
Jul 28 02:56:17.672160 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.672069    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-proxy-tls\") pod \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\" (UID: \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\") "
Jul 28 02:56:17.672160 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.672092    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbz6h\" (UniqueName: \"kubernetes.io/projected/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kube-api-access-kbz6h\") pod \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\" (UID: \"9de9f5b2-0547-4584-84c0-7de4b57bd2dc\") "
Jul 28 02:56:17.672882 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.672839    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9de9f5b2-0547-4584-84c0-7de4b57bd2dc" (UID: "9de9f5b2-0547-4584-84c0-7de4b57bd2dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:56:17.673023 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.672979    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9de9f5b2-0547-4584-84c0-7de4b57bd2dc" (UID: "9de9f5b2-0547-4584-84c0-7de4b57bd2dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:56:17.675724 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.675682    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kube-api-access-kbz6h" (OuterVolumeSpecName: "kube-api-access-kbz6h") pod "9de9f5b2-0547-4584-84c0-7de4b57bd2dc" (UID: "9de9f5b2-0547-4584-84c0-7de4b57bd2dc"). InnerVolumeSpecName "kube-api-access-kbz6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:56:17.676017 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.675977    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9de9f5b2-0547-4584-84c0-7de4b57bd2dc" (UID: "9de9f5b2-0547-4584-84c0-7de4b57bd2dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:56:17.761841 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.761818    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv_9de9f5b2-0547-4584-84c0-7de4b57bd2dc/storage-initializer/1.log"
Jul 28 02:56:17.761989 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.761905    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv" event={"ID":"9de9f5b2-0547-4584-84c0-7de4b57bd2dc","Type":"ContainerDied","Data":"d3c1ecf4a2dc5f952c485142ed866ece8ef9a57bc2ecd378db24f096d4533a36"}
Jul 28 02:56:17.761989 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.761923    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"
Jul 28 02:56:17.761989 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.761985    2555 scope.go:117] "RemoveContainer" containerID="4d7d5d3e252f91a682aa6263a9acc84bd6c87d7d9108df00f2e6cbfca4bb43ce"
Jul 28 02:56:17.772984 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.772965    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:56:17.773058 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.773006    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-proxy-tls\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:56:17.773058 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.773017    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbz6h\" (UniqueName: \"kubernetes.io/projected/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kube-api-access-kbz6h\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:56:17.773058 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.773025    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9de9f5b2-0547-4584-84c0-7de4b57bd2dc-kserve-provision-location\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:56:17.809445 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.809379    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"]
Jul 28 02:56:17.812937 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:17.812914    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-rjdjv"]
Jul 28 02:56:19.451882 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.451850    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9de9f5b2-0547-4584-84c0-7de4b57bd2dc" path="/var/lib/kubelet/pods/9de9f5b2-0547-4584-84c0-7de4b57bd2dc/volumes"
Jul 28 02:56:19.545170 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.545136    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rnhdv/must-gather-zrt6s"]
Jul 28 02:56:19.545378 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.545366    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kserve-container"
Jul 28 02:56:19.545431 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.545379    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kserve-container"
Jul 28 02:56:19.545431 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.545392    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="storage-initializer"
Jul 28 02:56:19.545431 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.545398    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="storage-initializer"
Jul 28 02:56:19.545431 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.545404    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9de9f5b2-0547-4584-84c0-7de4b57bd2dc" containerName="storage-initializer"
Jul 28 02:56:19.545431 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.545408    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9de9f5b2-0547-4584-84c0-7de4b57bd2dc" containerName="storage-initializer"
Jul 28 02:56:19.545431 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.545415    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9de9f5b2-0547-4584-84c0-7de4b57bd2dc" containerName="storage-initializer"
Jul 28 02:56:19.545431 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.545420    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9de9f5b2-0547-4584-84c0-7de4b57bd2dc" containerName="storage-initializer"
Jul 28 02:56:19.545431 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.545427    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kube-rbac-proxy"
Jul 28 02:56:19.545431 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.545431    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kube-rbac-proxy"
Jul 28 02:56:19.545770 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.545489    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9de9f5b2-0547-4584-84c0-7de4b57bd2dc" containerName="storage-initializer"
Jul 28 02:56:19.545770 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.545498    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kube-rbac-proxy"
Jul 28 02:56:19.545770 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.545503    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9de9f5b2-0547-4584-84c0-7de4b57bd2dc" containerName="storage-initializer"
Jul 28 02:56:19.545770 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.545508    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="726e7cda-079e-4c81-bb0f-68fc2cba10e7" containerName="kserve-container"
Jul 28 02:56:19.549533 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.549505    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rnhdv/must-gather-zrt6s"
Jul 28 02:56:19.552241 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.552223    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rnhdv\"/\"kube-root-ca.crt\""
Jul 28 02:56:19.552350 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.552311    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rnhdv\"/\"openshift-service-ca.crt\""
Jul 28 02:56:19.557238 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.557203    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rnhdv/must-gather-zrt6s"]
Jul 28 02:56:19.585473 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.585448    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47p57\" (UniqueName: \"kubernetes.io/projected/0614018c-e9b5-4506-9cf3-faf406b8a061-kube-api-access-47p57\") pod \"must-gather-zrt6s\" (UID: \"0614018c-e9b5-4506-9cf3-faf406b8a061\") " pod="openshift-must-gather-rnhdv/must-gather-zrt6s"
Jul 28 02:56:19.585594 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.585486    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0614018c-e9b5-4506-9cf3-faf406b8a061-must-gather-output\") pod \"must-gather-zrt6s\" (UID: \"0614018c-e9b5-4506-9cf3-faf406b8a061\") " pod="openshift-must-gather-rnhdv/must-gather-zrt6s"
Jul 28 02:56:19.686769 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.686741    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47p57\" (UniqueName: \"kubernetes.io/projected/0614018c-e9b5-4506-9cf3-faf406b8a061-kube-api-access-47p57\") pod \"must-gather-zrt6s\" (UID: \"0614018c-e9b5-4506-9cf3-faf406b8a061\") " pod="openshift-must-gather-rnhdv/must-gather-zrt6s"
Jul 28 02:56:19.686879 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.686779    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0614018c-e9b5-4506-9cf3-faf406b8a061-must-gather-output\") pod \"must-gather-zrt6s\" (UID: \"0614018c-e9b5-4506-9cf3-faf406b8a061\") " pod="openshift-must-gather-rnhdv/must-gather-zrt6s"
Jul 28 02:56:19.687143 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.687125    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0614018c-e9b5-4506-9cf3-faf406b8a061-must-gather-output\") pod \"must-gather-zrt6s\" (UID: \"0614018c-e9b5-4506-9cf3-faf406b8a061\") " pod="openshift-must-gather-rnhdv/must-gather-zrt6s"
Jul 28 02:56:19.695625 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.695605    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47p57\" (UniqueName: \"kubernetes.io/projected/0614018c-e9b5-4506-9cf3-faf406b8a061-kube-api-access-47p57\") pod \"must-gather-zrt6s\" (UID: \"0614018c-e9b5-4506-9cf3-faf406b8a061\") " pod="openshift-must-gather-rnhdv/must-gather-zrt6s"
Jul 28 02:56:19.858574 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.858536    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rnhdv/must-gather-zrt6s"
Jul 28 02:56:19.983165 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.983135    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rnhdv/must-gather-zrt6s"]
Jul 28 02:56:19.985862 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:56:19.985822    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0614018c_e9b5_4506_9cf3_faf406b8a061.slice/crio-06eff570057f45cd87fb1e5447e12086a7f38ddc44cbe8ccb371e577a6ec7f7f WatchSource:0}: Error finding container 06eff570057f45cd87fb1e5447e12086a7f38ddc44cbe8ccb371e577a6ec7f7f: Status 404 returned error can't find the container with id 06eff570057f45cd87fb1e5447e12086a7f38ddc44cbe8ccb371e577a6ec7f7f
Jul 28 02:56:19.987551 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:19.987507    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 02:56:20.776339 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:20.776273    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rnhdv/must-gather-zrt6s" event={"ID":"0614018c-e9b5-4506-9cf3-faf406b8a061","Type":"ContainerStarted","Data":"06eff570057f45cd87fb1e5447e12086a7f38ddc44cbe8ccb371e577a6ec7f7f"}
Jul 28 02:56:26.804416 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:26.804382    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rnhdv/must-gather-zrt6s" event={"ID":"0614018c-e9b5-4506-9cf3-faf406b8a061","Type":"ContainerStarted","Data":"3e4faf3955eb379deee6e474f325222246b3ed8cc9520547390bf2f29eedb735"}
Jul 28 02:56:26.804813 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:26.804425    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rnhdv/must-gather-zrt6s" event={"ID":"0614018c-e9b5-4506-9cf3-faf406b8a061","Type":"ContainerStarted","Data":"5e4c3c040881d989cc2a14586def9d190e414e10c0caf7522a8d850013d96c70"}
Jul 28 02:56:26.821146 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:26.821099    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rnhdv/must-gather-zrt6s" podStartSLOduration=1.7256533059999999 podStartE2EDuration="7.821084051s" totalImagesPullingTime="6.095430745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:56:26.82019385 +0000 UTC m=+5433.829712279" watchObservedRunningTime="2026-07-28 02:56:26.821084051 +0000 UTC m=+5433.830602470"
Jul 28 02:56:52.903906 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:52.903873    2555 generic.go:356] "Generic (PLEG): container finished" podID="0614018c-e9b5-4506-9cf3-faf406b8a061" containerID="5e4c3c040881d989cc2a14586def9d190e414e10c0caf7522a8d850013d96c70" exitCode=0
Jul 28 02:56:52.904383 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:52.903934    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rnhdv/must-gather-zrt6s" event={"ID":"0614018c-e9b5-4506-9cf3-faf406b8a061","Type":"ContainerDied","Data":"5e4c3c040881d989cc2a14586def9d190e414e10c0caf7522a8d850013d96c70"}
Jul 28 02:56:52.904383 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:52.904203    2555 scope.go:117] "RemoveContainer" containerID="5e4c3c040881d989cc2a14586def9d190e414e10c0caf7522a8d850013d96c70"
Jul 28 02:56:52.941914 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:52.941888    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rnhdv_must-gather-zrt6s_0614018c-e9b5-4506-9cf3-faf406b8a061/gather/0.log"
Jul 28 02:56:56.335988 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:56.335954    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f4xdf_be56d98e-e52b-4ea8-9187-511176665495/global-pull-secret-syncer/0.log"
Jul 28 02:56:56.498033 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:56.498005    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qpfgr_dff83d23-57b7-440c-b7a1-d2834e3d31c6/konnectivity-agent/0.log"
Jul 28 02:56:56.571504 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:56.571477    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-123.ec2.internal_3bb63b8ff0a229a53856e473495df153/haproxy/0.log"
Jul 28 02:56:58.451408 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.451373    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-rnhdv/must-gather-zrt6s"]
Jul 28 02:56:58.451847 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.451566    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-rnhdv/must-gather-zrt6s" podUID="0614018c-e9b5-4506-9cf3-faf406b8a061" containerName="copy" containerID="cri-o://3e4faf3955eb379deee6e474f325222246b3ed8cc9520547390bf2f29eedb735" gracePeriod=2
Jul 28 02:56:58.456171 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.456145    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-rnhdv/must-gather-zrt6s"]
Jul 28 02:56:58.681269 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.681246    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rnhdv_must-gather-zrt6s_0614018c-e9b5-4506-9cf3-faf406b8a061/copy/0.log"
Jul 28 02:56:58.682998 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.682978    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rnhdv/must-gather-zrt6s"
Jul 28 02:56:58.685303 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:56:58.685273    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zrt6s\" is forbidden: User \"system:node:ip-10-0-133-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rnhdv\": no relationship found between node 'ip-10-0-133-123.ec2.internal' and this object" podUID="0614018c-e9b5-4506-9cf3-faf406b8a061" pod="openshift-must-gather-rnhdv/must-gather-zrt6s"
Jul 28 02:56:58.787024 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.787000    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47p57\" (UniqueName: \"kubernetes.io/projected/0614018c-e9b5-4506-9cf3-faf406b8a061-kube-api-access-47p57\") pod \"0614018c-e9b5-4506-9cf3-faf406b8a061\" (UID: \"0614018c-e9b5-4506-9cf3-faf406b8a061\") "
Jul 28 02:56:58.787153 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.787056    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0614018c-e9b5-4506-9cf3-faf406b8a061-must-gather-output\") pod \"0614018c-e9b5-4506-9cf3-faf406b8a061\" (UID: \"0614018c-e9b5-4506-9cf3-faf406b8a061\") "
Jul 28 02:56:58.788574 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.788547    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0614018c-e9b5-4506-9cf3-faf406b8a061-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "0614018c-e9b5-4506-9cf3-faf406b8a061" (UID: "0614018c-e9b5-4506-9cf3-faf406b8a061"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:56:58.789217 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.789198    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0614018c-e9b5-4506-9cf3-faf406b8a061-kube-api-access-47p57" (OuterVolumeSpecName: "kube-api-access-47p57") pod "0614018c-e9b5-4506-9cf3-faf406b8a061" (UID: "0614018c-e9b5-4506-9cf3-faf406b8a061"). InnerVolumeSpecName "kube-api-access-47p57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:56:58.887838 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.887812    2555 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0614018c-e9b5-4506-9cf3-faf406b8a061-must-gather-output\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:56:58.887838 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.887835    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47p57\" (UniqueName: \"kubernetes.io/projected/0614018c-e9b5-4506-9cf3-faf406b8a061-kube-api-access-47p57\") on node \"ip-10-0-133-123.ec2.internal\" DevicePath \"\""
Jul 28 02:56:58.923472 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.923452    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rnhdv_must-gather-zrt6s_0614018c-e9b5-4506-9cf3-faf406b8a061/copy/0.log"
Jul 28 02:56:58.923819 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.923798    2555 generic.go:356] "Generic (PLEG): container finished" podID="0614018c-e9b5-4506-9cf3-faf406b8a061" containerID="3e4faf3955eb379deee6e474f325222246b3ed8cc9520547390bf2f29eedb735" exitCode=143
Jul 28 02:56:58.923887 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.923851    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rnhdv/must-gather-zrt6s"
Jul 28 02:56:58.923943 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.923910    2555 scope.go:117] "RemoveContainer" containerID="3e4faf3955eb379deee6e474f325222246b3ed8cc9520547390bf2f29eedb735"
Jul 28 02:56:58.926163 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:56:58.926137    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zrt6s\" is forbidden: User \"system:node:ip-10-0-133-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rnhdv\": no relationship found between node 'ip-10-0-133-123.ec2.internal' and this object" podUID="0614018c-e9b5-4506-9cf3-faf406b8a061" pod="openshift-must-gather-rnhdv/must-gather-zrt6s"
Jul 28 02:56:58.931398 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.931382    2555 scope.go:117] "RemoveContainer" containerID="5e4c3c040881d989cc2a14586def9d190e414e10c0caf7522a8d850013d96c70"
Jul 28 02:56:58.933646 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:56:58.933620    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zrt6s\" is forbidden: User \"system:node:ip-10-0-133-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rnhdv\": no relationship found between node 'ip-10-0-133-123.ec2.internal' and this object" podUID="0614018c-e9b5-4506-9cf3-faf406b8a061" pod="openshift-must-gather-rnhdv/must-gather-zrt6s"
Jul 28 02:56:58.942246 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.942230    2555 scope.go:117] "RemoveContainer" containerID="3e4faf3955eb379deee6e474f325222246b3ed8cc9520547390bf2f29eedb735"
Jul 28 02:56:58.942490 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:56:58.942469    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4faf3955eb379deee6e474f325222246b3ed8cc9520547390bf2f29eedb735\": container with ID starting with 3e4faf3955eb379deee6e474f325222246b3ed8cc9520547390bf2f29eedb735 not found: ID does not exist" containerID="3e4faf3955eb379deee6e474f325222246b3ed8cc9520547390bf2f29eedb735"
Jul 28 02:56:58.942577 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.942503    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4faf3955eb379deee6e474f325222246b3ed8cc9520547390bf2f29eedb735"} err="failed to get container status \"3e4faf3955eb379deee6e474f325222246b3ed8cc9520547390bf2f29eedb735\": rpc error: code = NotFound desc = could not find container \"3e4faf3955eb379deee6e474f325222246b3ed8cc9520547390bf2f29eedb735\": container with ID starting with 3e4faf3955eb379deee6e474f325222246b3ed8cc9520547390bf2f29eedb735 not found: ID does not exist"
Jul 28 02:56:58.942577 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.942543    2555 scope.go:117] "RemoveContainer" containerID="5e4c3c040881d989cc2a14586def9d190e414e10c0caf7522a8d850013d96c70"
Jul 28 02:56:58.942802 ip-10-0-133-123 kubenswrapper[2555]: E0728 02:56:58.942786    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4c3c040881d989cc2a14586def9d190e414e10c0caf7522a8d850013d96c70\": container with ID starting with 5e4c3c040881d989cc2a14586def9d190e414e10c0caf7522a8d850013d96c70 not found: ID does not exist" containerID="5e4c3c040881d989cc2a14586def9d190e414e10c0caf7522a8d850013d96c70"
Jul 28 02:56:58.942852 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:58.942806    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4c3c040881d989cc2a14586def9d190e414e10c0caf7522a8d850013d96c70"} err="failed to get container status \"5e4c3c040881d989cc2a14586def9d190e414e10c0caf7522a8d850013d96c70\": rpc error: code = NotFound desc = could not find container \"5e4c3c040881d989cc2a14586def9d190e414e10c0caf7522a8d850013d96c70\": container with ID starting with 5e4c3c040881d989cc2a14586def9d190e414e10c0caf7522a8d850013d96c70 not found: ID does not exist"
Jul 28 02:56:59.451467 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:56:59.451432    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0614018c-e9b5-4506-9cf3-faf406b8a061" path="/var/lib/kubelet/pods/0614018c-e9b5-4506-9cf3-faf406b8a061/volumes"
Jul 28 02:57:00.791010 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:00.790974    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m8x8k_ce30f949-8274-4b0f-b59c-91ab0f311dde/node-exporter/0.log"
Jul 28 02:57:00.813928 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:00.813908    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m8x8k_ce30f949-8274-4b0f-b59c-91ab0f311dde/kube-rbac-proxy/0.log"
Jul 28 02:57:00.835949 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:00.835925    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m8x8k_ce30f949-8274-4b0f-b59c-91ab0f311dde/init-textfile/0.log"
Jul 28 02:57:05.191845 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.191809    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"]
Jul 28 02:57:05.192211 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.192034    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0614018c-e9b5-4506-9cf3-faf406b8a061" containerName="gather"
Jul 28 02:57:05.192211 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.192044    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0614018c-e9b5-4506-9cf3-faf406b8a061" containerName="gather"
Jul 28 02:57:05.192211 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.192062    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0614018c-e9b5-4506-9cf3-faf406b8a061" containerName="copy"
Jul 28 02:57:05.192211 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.192069    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0614018c-e9b5-4506-9cf3-faf406b8a061" containerName="copy"
Jul 28 02:57:05.192211 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.192110    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0614018c-e9b5-4506-9cf3-faf406b8a061" containerName="gather"
Jul 28 02:57:05.192211 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.192117    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0614018c-e9b5-4506-9cf3-faf406b8a061" containerName="copy"
Jul 28 02:57:05.195453 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.195433    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.198138 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.198118    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wxmf9\"/\"openshift-service-ca.crt\""
Jul 28 02:57:05.198236 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.198134    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wxmf9\"/\"default-dockercfg-rxfnb\""
Jul 28 02:57:05.199840 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.199825    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wxmf9\"/\"kube-root-ca.crt\""
Jul 28 02:57:05.206784 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.206763    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"]
Jul 28 02:57:05.338127 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.338091    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e2645573-b4fb-4951-80e8-4d907d535951-podres\") pod \"perf-node-gather-daemonset-qsdn7\" (UID: \"e2645573-b4fb-4951-80e8-4d907d535951\") " pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.338127 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.338128    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2645573-b4fb-4951-80e8-4d907d535951-sys\") pod \"perf-node-gather-daemonset-qsdn7\" (UID: \"e2645573-b4fb-4951-80e8-4d907d535951\") " pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.338377 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.338162    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e2645573-b4fb-4951-80e8-4d907d535951-proc\") pod \"perf-node-gather-daemonset-qsdn7\" (UID: \"e2645573-b4fb-4951-80e8-4d907d535951\") " pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.338377 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.338224    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt778\" (UniqueName: \"kubernetes.io/projected/e2645573-b4fb-4951-80e8-4d907d535951-kube-api-access-tt778\") pod \"perf-node-gather-daemonset-qsdn7\" (UID: \"e2645573-b4fb-4951-80e8-4d907d535951\") " pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.338377 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.338268    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2645573-b4fb-4951-80e8-4d907d535951-lib-modules\") pod \"perf-node-gather-daemonset-qsdn7\" (UID: \"e2645573-b4fb-4951-80e8-4d907d535951\") " pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.439133 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.439096    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e2645573-b4fb-4951-80e8-4d907d535951-proc\") pod \"perf-node-gather-daemonset-qsdn7\" (UID: \"e2645573-b4fb-4951-80e8-4d907d535951\") " pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.439298 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.439159    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt778\" (UniqueName: \"kubernetes.io/projected/e2645573-b4fb-4951-80e8-4d907d535951-kube-api-access-tt778\") pod \"perf-node-gather-daemonset-qsdn7\" (UID: \"e2645573-b4fb-4951-80e8-4d907d535951\") " pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.439298 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.439184    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2645573-b4fb-4951-80e8-4d907d535951-lib-modules\") pod \"perf-node-gather-daemonset-qsdn7\" (UID: \"e2645573-b4fb-4951-80e8-4d907d535951\") " pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.439298 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.439215    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e2645573-b4fb-4951-80e8-4d907d535951-podres\") pod \"perf-node-gather-daemonset-qsdn7\" (UID: \"e2645573-b4fb-4951-80e8-4d907d535951\") " pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.439298 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.439236    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2645573-b4fb-4951-80e8-4d907d535951-sys\") pod \"perf-node-gather-daemonset-qsdn7\" (UID: \"e2645573-b4fb-4951-80e8-4d907d535951\") " pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.439479 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.439115    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e2645573-b4fb-4951-80e8-4d907d535951-proc\") pod \"perf-node-gather-daemonset-qsdn7\" (UID: \"e2645573-b4fb-4951-80e8-4d907d535951\") " pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.439479 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.439314    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2645573-b4fb-4951-80e8-4d907d535951-sys\") pod \"perf-node-gather-daemonset-qsdn7\" (UID: \"e2645573-b4fb-4951-80e8-4d907d535951\") " pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.439479 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.439313    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2645573-b4fb-4951-80e8-4d907d535951-lib-modules\") pod \"perf-node-gather-daemonset-qsdn7\" (UID: \"e2645573-b4fb-4951-80e8-4d907d535951\") " pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.439479 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.439310    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e2645573-b4fb-4951-80e8-4d907d535951-podres\") pod \"perf-node-gather-daemonset-qsdn7\" (UID: \"e2645573-b4fb-4951-80e8-4d907d535951\") " pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.447260 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.447192    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt778\" (UniqueName: \"kubernetes.io/projected/e2645573-b4fb-4951-80e8-4d907d535951-kube-api-access-tt778\") pod \"perf-node-gather-daemonset-qsdn7\" (UID: \"e2645573-b4fb-4951-80e8-4d907d535951\") " pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.504460 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.504439    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.544149 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.544120    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v6xwj_aa0f7d7b-c27c-476f-a48c-6a27b3057102/dns/0.log"
Jul 28 02:57:05.568632 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.568604    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v6xwj_aa0f7d7b-c27c-476f-a48c-6a27b3057102/kube-rbac-proxy/0.log"
Jul 28 02:57:05.592742 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.592720    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k5qjk_80b8c33c-c6e3-4540-ba14-4027f4147c7e/dns-node-resolver/0.log"
Jul 28 02:57:05.629940 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.629908    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"]
Jul 28 02:57:05.632997 ip-10-0-133-123 kubenswrapper[2555]: W0728 02:57:05.632973    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode2645573_b4fb_4951_80e8_4d907d535951.slice/crio-f8b70101d7fb09f92df7ed07e099f6673d6598b15dde57e916cbffe5b512417c WatchSource:0}: Error finding container f8b70101d7fb09f92df7ed07e099f6673d6598b15dde57e916cbffe5b512417c: Status 404 returned error can't find the container with id f8b70101d7fb09f92df7ed07e099f6673d6598b15dde57e916cbffe5b512417c
Jul 28 02:57:05.949829 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.949790    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7" event={"ID":"e2645573-b4fb-4951-80e8-4d907d535951","Type":"ContainerStarted","Data":"b32a980599f52e11fe504216567ec1aba900098588db7e83ff4fd3723740ce4b"}
Jul 28 02:57:05.950016 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.949855    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7" event={"ID":"e2645573-b4fb-4951-80e8-4d907d535951","Type":"ContainerStarted","Data":"f8b70101d7fb09f92df7ed07e099f6673d6598b15dde57e916cbffe5b512417c"}
Jul 28 02:57:05.950016 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.949894    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:05.964490 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:05.964440    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7" podStartSLOduration=0.964421565 podStartE2EDuration="964.421565ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 02:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:57:05.964297548 +0000 UTC m=+5472.973815973" watchObservedRunningTime="2026-07-28 02:57:05.964421565 +0000 UTC m=+5472.973939991"
Jul 28 02:57:06.258122 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:06.258092    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z9b5j_965314c0-5f5a-4d0e-a133-9fd61ae4c83c/node-ca/0.log"
Jul 28 02:57:07.293635 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:07.293604    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7kw4c_a37d3e7a-a5f2-465b-b814-31c0dc43842f/serve-healthcheck-canary/0.log"
Jul 28 02:57:07.748477 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:07.748453    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-88m5j_89dc4149-6058-4fe9-90e6-d4f5ef6dd651/kube-rbac-proxy/0.log"
Jul 28 02:57:07.770477 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:07.770456    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-88m5j_89dc4149-6058-4fe9-90e6-d4f5ef6dd651/exporter/0.log"
Jul 28 02:57:07.791724 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:07.791698    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-88m5j_89dc4149-6058-4fe9-90e6-d4f5ef6dd651/extractor/0.log"
Jul 28 02:57:10.344750 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:10.344720    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-sc9gn_9855f8c7-1a90-4a2b-90f9-e9c1fad8c1b3/server/0.log"
Jul 28 02:57:10.632573 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:10.632482    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-tvgjt_0d3eac21-7291-4d7e-8209-4d55b8a88f22/manager/0.log"
Jul 28 02:57:11.964422 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:11.964393    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wxmf9/perf-node-gather-daemonset-qsdn7"
Jul 28 02:57:14.844512 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:14.844481    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-66p5q_a594223b-e7f8-403e-92a9-272d7f200e4c/migrator/0.log"
Jul 28 02:57:14.867266 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:14.867247    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-66p5q_a594223b-e7f8-403e-92a9-272d7f200e4c/graceful-termination/0.log"
Jul 28 02:57:16.156152 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:16.156078    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9j9s8_2dd2db27-d5f8-4dab-a8dc-d75d4091a8fd/kube-multus/0.log"
Jul 28 02:57:16.529734 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:16.529709    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x8rd2_2ba62973-501c-45ad-9710-200326696aa3/kube-multus-additional-cni-plugins/0.log"
Jul 28 02:57:16.572668 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:16.572641    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x8rd2_2ba62973-501c-45ad-9710-200326696aa3/egress-router-binary-copy/0.log"
Jul 28 02:57:16.626135 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:16.626105    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x8rd2_2ba62973-501c-45ad-9710-200326696aa3/cni-plugins/0.log"
Jul 28 02:57:16.685748 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:16.685720    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x8rd2_2ba62973-501c-45ad-9710-200326696aa3/bond-cni-plugin/0.log"
Jul 28 02:57:16.706028 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:16.706006    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x8rd2_2ba62973-501c-45ad-9710-200326696aa3/routeoverride-cni/0.log"
Jul 28 02:57:16.727953 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:16.727930    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x8rd2_2ba62973-501c-45ad-9710-200326696aa3/whereabouts-cni-bincopy/0.log"
Jul 28 02:57:16.750626 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:16.750602    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x8rd2_2ba62973-501c-45ad-9710-200326696aa3/whereabouts-cni/0.log"
Jul 28 02:57:16.889120 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:16.889045    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-k24x4_3442c3b8-9ee2-4ad0-bbcf-b0918932580e/network-metrics-daemon/0.log"
Jul 28 02:57:16.913226 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:16.913201    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-k24x4_3442c3b8-9ee2-4ad0-bbcf-b0918932580e/kube-rbac-proxy/0.log"
Jul 28 02:57:17.959941 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:17.959905    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54bw2_b9607f9f-591a-4033-83ec-44deaee179e6/ovn-controller/0.log"
Jul 28 02:57:17.979476 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:17.979447    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54bw2_b9607f9f-591a-4033-83ec-44deaee179e6/ovn-acl-logging/0.log"
Jul 28 02:57:18.003021 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:18.002994    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54bw2_b9607f9f-591a-4033-83ec-44deaee179e6/kube-rbac-proxy-node/0.log"
Jul 28 02:57:18.024321 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:18.024301    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54bw2_b9607f9f-591a-4033-83ec-44deaee179e6/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 02:57:18.047818 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:18.047799    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54bw2_b9607f9f-591a-4033-83ec-44deaee179e6/northd/0.log"
Jul 28 02:57:18.069852 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:18.069832    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54bw2_b9607f9f-591a-4033-83ec-44deaee179e6/nbdb/0.log"
Jul 28 02:57:18.090873 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:18.090854    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54bw2_b9607f9f-591a-4033-83ec-44deaee179e6/sbdb/0.log"
Jul 28 02:57:18.145531 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:18.145494    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54bw2_b9607f9f-591a-4033-83ec-44deaee179e6/ovnkube-controller/0.log"
Jul 28 02:57:19.419111 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:19.419086    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-q8gnk_4ba73e34-6176-472c-bbde-ffd0c9b335eb/network-check-target-container/0.log"
Jul 28 02:57:20.402321 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:20.402297    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-v7xpp_6f547cf9-cb20-4d9b-8c20-1f66359d6103/iptables-alerter/0.log"
Jul 28 02:57:21.047585 ip-10-0-133-123 kubenswrapper[2555]: I0728 02:57:21.047560    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-pq94g_82b589cd-bf47-4440-a8dd-5f5cbf9794cb/tuned/0.log"