Apr 16 16:29:50.468816 ip-10-0-140-164 systemd[1]: Starting Kubernetes Kubelet...
Apr 16 16:29:50.972632 ip-10-0-140-164 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 16 16:29:50.972632 ip-10-0-140-164 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Apr 16 16:29:50.972632 ip-10-0-140-164 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 16 16:29:50.972632 ip-10-0-140-164 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Apr 16 16:29:50.972632 ip-10-0-140-164 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 16 16:29:50.975914 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.975838    2578 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Apr 16 16:29:50.978906 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978891    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 16 16:29:50.978906 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978905    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978910    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978914    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978917    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978921    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978923    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978926    2578 feature_gate.go:328] unrecognized feature gate: Example2
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978929    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978932    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978934    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978937    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978940    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978942    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978945    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978948    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978951    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978953    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978956    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978959    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978962    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 16 16:29:50.978967 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978965    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978967    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978970    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978973    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978976    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978979    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978982    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978985    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978987    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978990    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978994    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.978998    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979001    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979004    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979006    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979009    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979012    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979015    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979017    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 16 16:29:50.979438 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979020    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979023    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979025    2578 feature_gate.go:328] unrecognized feature gate: Example
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979028    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979031    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979034    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979037    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979039    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979042    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979044    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979047    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979049    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979052    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979054    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979057    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979061    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979064    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979066    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979069    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979072    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 16 16:29:50.979952 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979074    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979077    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979080    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979083    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979085    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979088    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979091    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979094    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979097    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979100    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979103    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979105    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979108    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979112    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979116    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979119    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979121    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979124    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979127    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 16 16:29:50.980504 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979129    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979132    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979134    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979137    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979140    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979143    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979146    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979536    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979543    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979547    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979564    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979567    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979570    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979573    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979576    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979579    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979581    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979584    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979586    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979590    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 16 16:29:50.980982 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979593    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979595    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979598    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979600    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979603    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979606    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979608    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979611    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979614    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979616    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979619    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979621    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979625    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979629    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979631    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979634    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979636    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979639    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979642    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979645    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 16 16:29:50.981486 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979648    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979651    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979654    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979656    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979659    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979661    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979664    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979667    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979669    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979672    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979675    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979678    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979681    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979683    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979686    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979688    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979691    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979694    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979696    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979698    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 16 16:29:50.982026 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979701    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979704    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979706    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979709    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979712    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979714    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979716    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979719    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979721    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979726    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979730    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979733    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979737    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979740    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979744    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979746    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979749    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979752    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979755    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 16 16:29:50.982509 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979758    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979760    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979763    2578 feature_gate.go:328] unrecognized feature gate: Example2
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979765    2578 feature_gate.go:328] unrecognized feature gate: Example
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979768    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979770    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979774    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979776    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979779    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979782    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979784    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979787    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979789    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.979792    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980690    2578 flags.go:64] FLAG: --address="0.0.0.0"
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980700    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980706    2578 flags.go:64] FLAG: --anonymous-auth="true"
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980710    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980715    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980718    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980723    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Apr 16 16:29:50.982983 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980727    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980731    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980734    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980738    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980742    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980745    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980748    2578 flags.go:64] FLAG: --cgroup-root=""
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980751    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980754    2578 flags.go:64] FLAG: --client-ca-file=""
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980757    2578 flags.go:64] FLAG: --cloud-config=""
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980760    2578 flags.go:64] FLAG: --cloud-provider="external"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980763    2578 flags.go:64] FLAG: --cluster-dns="[]"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980767    2578 flags.go:64] FLAG: --cluster-domain=""
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980770    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980773    2578 flags.go:64] FLAG: --config-dir=""
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980776    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980780    2578 flags.go:64] FLAG: --container-log-max-files="5"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980784    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980787    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980790    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980793    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980796    2578 flags.go:64] FLAG: --contention-profiling="false"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980799    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980802    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980805    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Apr 16 16:29:50.983499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980808    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980813    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980816    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980819    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980822    2578 flags.go:64] FLAG: --enable-load-reader="false"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980825    2578 flags.go:64] FLAG: --enable-server="true"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980827    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980831    2578 flags.go:64] FLAG: --event-burst="100"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980834    2578 flags.go:64] FLAG: --event-qps="50"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980838    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980841    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980844    2578 flags.go:64] FLAG: --eviction-hard=""
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980848    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980851    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980854    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980858    2578 flags.go:64] FLAG: --eviction-soft=""
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980861    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980864    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980867    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980870    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980873    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980875    2578 flags.go:64] FLAG: --fail-swap-on="true"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980878    2578 flags.go:64] FLAG: --feature-gates=""
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980882    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980885    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Apr 16 16:29:50.984122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980889    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980892    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980896    2578 flags.go:64] FLAG: --healthz-port="10248"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980899    2578 flags.go:64] FLAG: --help="false"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980902    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-140-164.ec2.internal"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980905    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980909    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980912    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980915    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980918    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980922    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980925    2578 flags.go:64] FLAG: --image-service-endpoint=""
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980927    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980930    2578 flags.go:64] FLAG: --kube-api-burst="100"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980940    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980945    2578 flags.go:64] FLAG: --kube-api-qps="50"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980948    2578 flags.go:64] FLAG: --kube-reserved=""
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980952    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980955    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980958    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980961    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980964    2578 flags.go:64] FLAG: --lock-file=""
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980967    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980970    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Apr 16 16:29:50.984741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980973    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980983    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980987    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980990    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980993    2578 flags.go:64] FLAG: --logging-format="text"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980996    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.980999    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981002    2578 flags.go:64] FLAG: --manifest-url=""
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981005    2578 flags.go:64] FLAG: --manifest-url-header=""
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981010    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981014    2578 flags.go:64] FLAG: --max-open-files="1000000"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981018    2578 flags.go:64] FLAG: --max-pods="110"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981021    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981024    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981026    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981030    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981033    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981036    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981038    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981046    2578 flags.go:64] FLAG: --node-status-max-images="50"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981049    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981053    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981056    2578 flags.go:64] FLAG: --pod-cidr=""
Apr 16 16:29:50.985331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981058    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dc76bab72f320de3d4105c90d73c4fb139c09e20ce0fa8dcbc0cb59920d27dec"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981065    2578 flags.go:64] FLAG: --pod-manifest-path=""
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981068    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981071    2578 flags.go:64] FLAG: --pods-per-core="0"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981074    2578 flags.go:64] FLAG: --port="10250"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981078    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981080    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bf62f5be8911f4a2"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981084    2578 flags.go:64] FLAG: --qos-reserved=""
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981087    2578 flags.go:64] FLAG: --read-only-port="10255"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981090    2578 flags.go:64] FLAG: --register-node="true"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981093    2578 flags.go:64] FLAG: --register-schedulable="true"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981096    2578 flags.go:64] FLAG: --register-with-taints=""
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981100    2578 flags.go:64] FLAG: --registry-burst="10"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981102    2578 flags.go:64] FLAG: --registry-qps="5"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981105    2578 flags.go:64] FLAG: --reserved-cpus=""
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981108    2578 flags.go:64] FLAG: --reserved-memory=""
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981112    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981115    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981118    2578 flags.go:64] FLAG: --rotate-certificates="false"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981121    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981124    2578 flags.go:64] FLAG: --runonce="false"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981127    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981130    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981133    2578 flags.go:64] FLAG: --seccomp-default="false"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981136    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981139    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Apr 16 16:29:50.985915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981142    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981146    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981149    2578 flags.go:64] FLAG: --storage-driver-password="root"
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981152    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981154    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981157    2578 flags.go:64] FLAG: --storage-driver-user="root"
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981160    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981163    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981166    2578 flags.go:64] FLAG: --system-cgroups=""
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981169    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981178    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981181    2578 flags.go:64] FLAG: --tls-cert-file=""
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981184    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981189    2578 flags.go:64] FLAG: --tls-min-version=""
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981192    2578 flags.go:64] FLAG: --tls-private-key-file=""
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981195    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981198    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981200    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981203    2578 flags.go:64] FLAG: --v="2"
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981207    2578 flags.go:64] FLAG: --version="false"
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981211    2578 flags.go:64] FLAG: --vmodule=""
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981215    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.981219    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981313    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 16 16:29:50.986535 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981317    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981320    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981323    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981326    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981329    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981331    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981334    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981337    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981340    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981342    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981345    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981347    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981350    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981353    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981355    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981358    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981361    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981364    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981368    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 16 16:29:50.987180 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981371    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981374    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981377    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981380    2578 feature_gate.go:328] unrecognized feature gate: Example2
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981383    2578 feature_gate.go:328] unrecognized feature gate: Example
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981385    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981388    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981390    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981393    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981396    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981398    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981401    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981403    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981406    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981408    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981411    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981414    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981417    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981420    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981423    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 16 16:29:50.987710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981425    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981428    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981430    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981433    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981435    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981438    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981440    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981443    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981446    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981449    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981451    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981457    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981460    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981463    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981466    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981469    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981473    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981477    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981480    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 16 16:29:50.988202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981482    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981485    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981488    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981491    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981493    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981496    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981499    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981501    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981504    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981506    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981509    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981512    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981515    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981518    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981520    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981523    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981526    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981528    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981531    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981533    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 16 16:29:50.988681 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981536    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 16 16:29:50.989181 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981540    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 16 16:29:50.989181 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981543    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 16 16:29:50.989181 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981545    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 16 16:29:50.989181 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981549    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 16 16:29:50.989181 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981566    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 16 16:29:50.989181 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.981569    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 16 16:29:50.989181 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.982207    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 16 16:29:50.989181 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.988224    2578 server.go:530] "Kubelet version" kubeletVersion="v1.33.9"
Apr 16 16:29:50.989181 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.988240    2578 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Apr 16 16:29:50.989181 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988285    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 16 16:29:50.989181 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988290    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 16 16:29:50.989181 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988293    2578 feature_gate.go:328] unrecognized feature gate: Example
Apr 16 16:29:50.989181 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988296    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 16 16:29:50.989181 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988299    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 16 16:29:50.989181 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988304    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988308    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988311    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988314    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988317    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988320    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988322    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988325    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988329    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988331    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988335    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988338    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988340    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988344    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988347    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988350    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988352    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988355    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988358    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988360    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 16 16:29:50.989581 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988363    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988366    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988369    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988371    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988374    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988377    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988380    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988383    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988386    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988388    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988391    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988394    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988397    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988400    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988402    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988405    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988407    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988410    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988413    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 16 16:29:50.990076 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988415    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988418    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988422    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988424    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988427    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988429    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988432    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988435    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988437    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988440    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988442    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988445    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988448    2578 feature_gate.go:328] unrecognized feature gate: Example2
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988451    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988455    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988458    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988461    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988463    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988467    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988470    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 16 16:29:50.990541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988473    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988475    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988479    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988481    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988484    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988486    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988489    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988491    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988494    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988496    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988499    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988501    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988504    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988506    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988509    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988512    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988514    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988517    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988519    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 16 16:29:50.991046 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988522    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 16 16:29:50.991541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988525    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 16 16:29:50.991541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988527    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 16 16:29:50.991541 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.988532    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 16 16:29:50.991541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988645    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 16 16:29:50.991541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988650    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 16 16:29:50.991541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988653    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 16 16:29:50.991541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988656    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 16 16:29:50.991541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988659    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 16 16:29:50.991541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988662    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 16 16:29:50.991541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988666    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 16 16:29:50.991541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988669    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 16 16:29:50.991541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988672    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 16 16:29:50.991541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988675    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 16 16:29:50.991541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988677    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 16 16:29:50.991541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988680    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 16 16:29:50.991541 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988683    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988685    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988688    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988691    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988694    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988697    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988699    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988702    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988705    2578 feature_gate.go:328] unrecognized feature gate: Example2
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988707    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988710    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988712    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988715    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988718    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988721    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988723    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988726    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988728    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988731    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988734    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 16 16:29:50.991954 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988736    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988739    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988741    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988744    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988747    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988749    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988752    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988755    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988758    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988760    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988763    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988765    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988768    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988770    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988773    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988776    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988778    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988781    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988783    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988786    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 16 16:29:50.992437 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988788    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988791    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988793    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988796    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988798    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988801    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988804    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988806    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988809    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988812    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988814    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988817    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988819    2578 feature_gate.go:328] unrecognized feature gate: Example
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988822    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988824    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988827    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988829    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988832    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988835    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 16 16:29:50.992953 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988837    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 16 16:29:50.993412 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988840    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 16 16:29:50.993412 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988843    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 16 16:29:50.993412 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988845    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 16 16:29:50.993412 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988848    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 16 16:29:50.993412 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988851    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 16 16:29:50.993412 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988853    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 16 16:29:50.993412 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988856    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 16 16:29:50.993412 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988858    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 16 16:29:50.993412 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988861    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 16 16:29:50.993412 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988863    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 16 16:29:50.993412 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988867    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 16 16:29:50.993412 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988871    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 16 16:29:50.993412 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988874    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 16 16:29:50.993412 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:50.988876    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 16 16:29:50.993412 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.988881    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 16 16:29:50.993798 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.989693    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Apr 16 16:29:50.993798 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.993752    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Apr 16 16:29:50.994747 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.994733    2578 server.go:1019] "Starting client certificate rotation"
Apr 16 16:29:50.995211 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.995189    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Apr 16 16:29:50.995271 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:50.995244    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Apr 16 16:29:51.027114 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.027094    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Apr 16 16:29:51.033994 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.033973    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Apr 16 16:29:51.046915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.046899    2578 log.go:25] "Validated CRI v1 runtime API"
Apr 16 16:29:51.054187 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.054168    2578 log.go:25] "Validated CRI v1 image API"
Apr 16 16:29:51.057851 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.057835    2578 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Apr 16 16:29:51.058191 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.058177    2578 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Apr 16 16:29:51.062339 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.062318    2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8ae1e038-5c22-4a91-a221-534573670a22:/dev/nvme0n1p4 b7cf0c5c-f259-4035-bfee-7fee7a892731:/dev/nvme0n1p3]
Apr 16 16:29:51.062382 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.062340    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Apr 16 16:29:51.068409 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.068305    2578 manager.go:217] Machine: {Timestamp:2026-04-16 16:29:51.066049476 +0000 UTC m=+0.463265873 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100886 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2daf88e22217968f8b0380245bcb45 SystemUUID:ec2daf88-e222-1796-8f8b-0380245bcb45 BootID:632d3fc0-a090-4430-9044-074bcecd8cfb Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6098944 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:27:67:5b:a4:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:27:67:5b:a4:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:3e:38:ad:c9:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Apr 16 16:29:51.068409 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.068405    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Apr 16 16:29:51.068514 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.068502    2578 manager.go:233] Version: {KernelVersion:5.14.0-570.104.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260401-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Apr 16 16:29:51.070811 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.070785    2578 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Apr 16 16:29:51.070939 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.070815    2578 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-164.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Apr 16 16:29:51.070983 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.070945    2578 topology_manager.go:138] "Creating topology manager with none policy"
Apr 16 16:29:51.070983 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.070954    2578 container_manager_linux.go:306] "Creating device plugin manager"
Apr 16 16:29:51.070983 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.070967    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Apr 16 16:29:51.072049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.072038    2578 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Apr 16 16:29:51.072975 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.072965    2578 state_mem.go:36] "Initialized new in-memory state store"
Apr 16 16:29:51.073080 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.073071    2578 server.go:1267] "Using root directory" path="/var/lib/kubelet"
Apr 16 16:29:51.075670 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.075661    2578 kubelet.go:491] "Attempting to sync node with API server"
Apr 16 16:29:51.075706 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.075679    2578 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
Apr 16 16:29:51.075706 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.075690    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Apr 16 16:29:51.075706 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.075698    2578 kubelet.go:397] "Adding apiserver pod source"
Apr 16 16:29:51.075706 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.075706    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Apr 16 16:29:51.076992 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.076980    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Apr 16 16:29:51.077047 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.076999    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Apr 16 16:29:51.077355 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.077338    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vhjr4"
Apr 16 16:29:51.080191 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.080176    2578 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.10-2.rhaos4.20.gita4d0894.el9" apiVersion="v1"
Apr 16 16:29:51.081889 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.081875    2578 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Apr 16 16:29:51.083228 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.083217    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Apr 16 16:29:51.083284 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.083233    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Apr 16 16:29:51.083284 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.083239    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Apr 16 16:29:51.083284 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.083244    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Apr 16 16:29:51.083284 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.083249    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Apr 16 16:29:51.083284 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.083262    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Apr 16 16:29:51.083284 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.083268    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Apr 16 16:29:51.083284 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.083274    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Apr 16 16:29:51.083284 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.083280    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Apr 16 16:29:51.083284 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.083287    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Apr 16 16:29:51.083522 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.083299    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Apr 16 16:29:51.084115 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.084105    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Apr 16 16:29:51.085720 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.085704    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vhjr4"
Apr 16 16:29:51.086372 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.086360    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Apr 16 16:29:51.086372 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.086372    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
Apr 16 16:29:51.086546 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.086525    2578 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Apr 16 16:29:51.086604 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.086527    2578 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-164.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Apr 16 16:29:51.092007 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.091986    2578 watchdog_linux.go:99] "Systemd watchdog is not enabled"
Apr 16 16:29:51.092099 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.092031    2578 server.go:1295] "Started kubelet"
Apr 16 16:29:51.092282 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.092237    2578 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Apr 16 16:29:51.092338 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.092304    2578 server_v1.go:47] "podresources" method="list" useActivePods=true
Apr 16 16:29:51.092526 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.092501    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Apr 16 16:29:51.092840 ip-10-0-140-164 systemd[1]: Started Kubernetes Kubelet.
Apr 16 16:29:51.093688 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.093672    2578 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Apr 16 16:29:51.094276 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.094259    2578 server.go:317] "Adding debug handlers to kubelet server"
Apr 16 16:29:51.099540 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.099518    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Apr 16 16:29:51.100270 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.100254    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Apr 16 16:29:51.103866 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.102129    2578 factory.go:55] Registering systemd factory
Apr 16 16:29:51.103866 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.102280    2578 factory.go:223] Registration of the systemd container factory successfully
Apr 16 16:29:51.103866 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.103027    2578 factory.go:153] Registering CRI-O factory
Apr 16 16:29:51.103866 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.103043    2578 factory.go:223] Registration of the crio container factory successfully
Apr 16 16:29:51.103866 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.103095    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-164.ec2.internal\" not found"
Apr 16 16:29:51.103866 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.103112    2578 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
Apr 16 16:29:51.103866 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.103131    2578 volume_manager.go:295] "The desired_state_of_world populator starts"
Apr 16 16:29:51.103866 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.103144    2578 volume_manager.go:297] "Starting Kubelet Volume Manager"
Apr 16 16:29:51.103866 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.103147    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Apr 16 16:29:51.103866 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.103166    2578 factory.go:103] Registering Raw factory
Apr 16 16:29:51.103866 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.103179    2578 manager.go:1196] Started watching for new ooms in manager
Apr 16 16:29:51.103866 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.103280    2578 reconstruct.go:97] "Volume reconstruction finished"
Apr 16 16:29:51.103866 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.103290    2578 reconciler.go:26] "Reconciler: start to sync state"
Apr 16 16:29:51.103866 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.103589    2578 manager.go:319] Starting recovery of all containers
Apr 16 16:29:51.104669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.104508    2578 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 16 16:29:51.107176 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.107135    2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-164.ec2.internal\" not found" node="ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.110953 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.110934    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-164.ec2.internal" not found
Apr 16 16:29:51.113717 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.113704    2578 manager.go:324] Recovery completed
Apr 16 16:29:51.117477 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.117466    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 16 16:29:51.119711 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.119696    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasSufficientMemory"
Apr 16 16:29:51.119782 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.119722    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasNoDiskPressure"
Apr 16 16:29:51.119782 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.119732    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasSufficientPID"
Apr 16 16:29:51.120164 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.120151    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Apr 16 16:29:51.120220 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.120164    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Apr 16 16:29:51.120220 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.120182    2578 state_mem.go:36] "Initialized new in-memory state store"
Apr 16 16:29:51.122806 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.122793    2578 policy_none.go:49] "None policy: Start"
Apr 16 16:29:51.122861 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.122809    2578 memory_manager.go:186] "Starting memorymanager" policy="None"
Apr 16 16:29:51.122861 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.122820    2578 state_mem.go:35] "Initializing new in-memory state store"
Apr 16 16:29:51.126640 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.126627    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-164.ec2.internal" not found
Apr 16 16:29:51.170103 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.170088    2578 manager.go:341] "Starting Device Plugin manager"
Apr 16 16:29:51.170199 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.170121    2578 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Apr 16 16:29:51.170199 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.170131    2578 server.go:85] "Starting device plugin registration server"
Apr 16 16:29:51.170368 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.170356    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Apr 16 16:29:51.170428 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.170370    2578 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Apr 16 16:29:51.170488 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.170455    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Apr 16 16:29:51.170566 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.170543    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Apr 16 16:29:51.170621 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.170578    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Apr 16 16:29:51.170993 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.170974    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Apr 16 16:29:51.171071 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.171014    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-164.ec2.internal\" not found"
Apr 16 16:29:51.185622 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.185601    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-164.ec2.internal" not found
Apr 16 16:29:51.229375 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.229325    2578 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
Apr 16 16:29:51.230482 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.230464    2578 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
Apr 16 16:29:51.230583 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.230489    2578 status_manager.go:230] "Starting to sync pod status with apiserver"
Apr 16 16:29:51.230583 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.230504    2578 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Apr 16 16:29:51.230583 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.230510    2578 kubelet.go:2451] "Starting kubelet main sync loop"
Apr 16 16:29:51.230583 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.230544    2578 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Apr 16 16:29:51.234517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.234501    2578 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 16 16:29:51.271215 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.271195    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 16 16:29:51.272224 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.272207    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasSufficientMemory"
Apr 16 16:29:51.272324 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.272238    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasNoDiskPressure"
Apr 16 16:29:51.272324 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.272252    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasSufficientPID"
Apr 16 16:29:51.272324 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.272277    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.282501 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.282481    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.282615 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.282502    2578 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-164.ec2.internal\": node \"ip-10-0-140-164.ec2.internal\" not found"
Apr 16 16:29:51.311608 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.311590    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-164.ec2.internal\" not found"
Apr 16 16:29:51.330605 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.330580    2578 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-164.ec2.internal"]
Apr 16 16:29:51.330659 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.330654    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 16 16:29:51.333621 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.333608    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasSufficientMemory"
Apr 16 16:29:51.333679 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.333631    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasNoDiskPressure"
Apr 16 16:29:51.333679 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.333640    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasSufficientPID"
Apr 16 16:29:51.334777 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.334766    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 16 16:29:51.334910 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.334890    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.334945 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.334916    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 16 16:29:51.337105 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.337089    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasSufficientMemory"
Apr 16 16:29:51.337105 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.337103    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasSufficientMemory"
Apr 16 16:29:51.337228 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.337117    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasNoDiskPressure"
Apr 16 16:29:51.337228 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.337124    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasNoDiskPressure"
Apr 16 16:29:51.337228 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.337129    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasSufficientPID"
Apr 16 16:29:51.337228 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.337140    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasSufficientPID"
Apr 16 16:29:51.338707 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.338691    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.338787 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.338717    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 16 16:29:51.341140 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.341123    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasSufficientMemory"
Apr 16 16:29:51.341216 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.341150    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasNoDiskPressure"
Apr 16 16:29:51.341216 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.341159    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeHasSufficientPID"
Apr 16 16:29:51.368047 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.368025    2578 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-164.ec2.internal\" not found" node="ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.372025 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.372009    2578 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-164.ec2.internal\" not found" node="ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.404438 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.404414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3a26939a85c3397690ad7fddb03cc34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal\" (UID: \"c3a26939a85c3397690ad7fddb03cc34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.404438 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.404437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3a26939a85c3397690ad7fddb03cc34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal\" (UID: \"c3a26939a85c3397690ad7fddb03cc34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.404576 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.404452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0a4ffc4283ad098ce5617bb701ba0817-config\") pod \"kube-apiserver-proxy-ip-10-0-140-164.ec2.internal\" (UID: \"0a4ffc4283ad098ce5617bb701ba0817\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.411932 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.411911    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-164.ec2.internal\" not found"
Apr 16 16:29:51.505075 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.505030    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3a26939a85c3397690ad7fddb03cc34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal\" (UID: \"c3a26939a85c3397690ad7fddb03cc34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.505075 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.505055    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0a4ffc4283ad098ce5617bb701ba0817-config\") pod \"kube-apiserver-proxy-ip-10-0-140-164.ec2.internal\" (UID: \"0a4ffc4283ad098ce5617bb701ba0817\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.505075 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.505072    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3a26939a85c3397690ad7fddb03cc34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal\" (UID: \"c3a26939a85c3397690ad7fddb03cc34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.505238 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.505126    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3a26939a85c3397690ad7fddb03cc34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal\" (UID: \"c3a26939a85c3397690ad7fddb03cc34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.505238 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.505142    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0a4ffc4283ad098ce5617bb701ba0817-config\") pod \"kube-apiserver-proxy-ip-10-0-140-164.ec2.internal\" (UID: \"0a4ffc4283ad098ce5617bb701ba0817\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.505238 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.505143    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3a26939a85c3397690ad7fddb03cc34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal\" (UID: \"c3a26939a85c3397690ad7fddb03cc34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.512139 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.512122    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-164.ec2.internal\" not found"
Apr 16 16:29:51.612975 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.612949    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-164.ec2.internal\" not found"
Apr 16 16:29:51.671124 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.671104    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.674615 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.674603    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.ec2.internal"
Apr 16 16:29:51.713867 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.713848    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-164.ec2.internal\" not found"
Apr 16 16:29:51.814453 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.814406    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-164.ec2.internal\" not found"
Apr 16 16:29:51.914951 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:51.914930    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-164.ec2.internal\" not found"
Apr 16 16:29:51.994232 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.994213    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Apr 16 16:29:51.994753 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.994341    2578 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Apr 16 16:29:51.994753 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:51.994352    2578 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Apr 16 16:29:52.015771 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:52.015745    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-164.ec2.internal\" not found"
Apr 16 16:29:52.027522 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.027505    2578 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 16 16:29:52.076139 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.076121    2578 apiserver.go:52] "Watching apiserver"
Apr 16 16:29:52.085032 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.085014    2578 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Apr 16 16:29:52.085438 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.085413    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-fbgtp","openshift-multus/multus-7cqz2","openshift-multus/multus-additional-cni-plugins-jnptn","openshift-multus/network-metrics-daemon-wq65n","openshift-network-diagnostics/network-check-target-w2mqw","openshift-network-operator/iptables-alerter-xdwz4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns","openshift-cluster-node-tuning-operator/tuned-pdmjc","openshift-image-registry/node-ca-7qft9","openshift-ovn-kubernetes/ovnkube-node-wd8ld","kube-system/konnectivity-agent-rpnzb"]
Apr 16 16:29:52.087179 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.087165    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fbgtp"
Apr 16 16:29:52.088221 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.088204    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.089203 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.089177    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Apr 16 16:29:52.089296 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.089205    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wpjpl\""
Apr 16 16:29:52.089296 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.089225    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Apr 16 16:29:52.090087 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.090074    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Apr 16 16:29:52.090276 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.090259    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.090401 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.090386    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:29:52.090475 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:52.090455    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:29:52.091043 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.090975    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Apr 16 16:29:52.091043 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.091016    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rvtq8\""
Apr 16 16:29:52.091450 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.091401    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-04-15 16:24:51 +0000 UTC" deadline="2028-01-30 22:44:07.969562266 +0000 UTC"
Apr 16 16:29:52.091450 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.091446    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Apr 16 16:29:52.091936 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.091918    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Apr 16 16:29:52.092011 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.091449    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15702h14m15.878117369s"
Apr 16 16:29:52.092825 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.092800    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Apr 16 16:29:52.093041 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.092972    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Apr 16 16:29:52.093041 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.093037    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qrl64\""
Apr 16 16:29:52.093144 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.093091    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xdwz4"
Apr 16 16:29:52.093192 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.093184    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:29:52.093322 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:52.093271    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:29:52.094626 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.094583    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.095309 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.095288    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4wxsw\""
Apr 16 16:29:52.095402 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.095380    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Apr 16 16:29:52.095466 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.095385    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Apr 16 16:29:52.095466 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.095437    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Apr 16 16:29:52.095830 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.095811    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.096249 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.096229    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Apr 16 16:29:52.096527 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.096514    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Apr 16 16:29:52.096860 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.096845    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Apr 16 16:29:52.096961 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.096941    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7qft9"
Apr 16 16:29:52.097025 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.096976    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-82746\""
Apr 16 16:29:52.097931 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.097913    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Apr 16 16:29:52.098104 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.098090    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gl475\""
Apr 16 16:29:52.098256 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.098174    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Apr 16 16:29:52.098409 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.098389    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.099195 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.099177    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Apr 16 16:29:52.099281 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.099200    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Apr 16 16:29:52.099281 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.099224    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5p66s\""
Apr 16 16:29:52.099281 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.099177    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Apr 16 16:29:52.099617 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.099602    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Apr 16 16:29:52.099753 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.099732    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rpnzb"
Apr 16 16:29:52.100244 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.100229    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Apr 16 16:29:52.100640 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.100618    2578 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal"
Apr 16 16:29:52.100738 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.100640    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Apr 16 16:29:52.100738 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.100651    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Apr 16 16:29:52.100738 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.100624    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Apr 16 16:29:52.100899 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.100620    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Apr 16 16:29:52.100958 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.100904    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w97np\""
Apr 16 16:29:52.101015 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.100999    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Apr 16 16:29:52.101783 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.101613    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Apr 16 16:29:52.101783 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.101613    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cqrjn\""
Apr 16 16:29:52.101783 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.101660    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Apr 16 16:29:52.103799 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.103784    2578 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
Apr 16 16:29:52.109662 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.109646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-run\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.109745 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.109669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-slash\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.109745 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.109685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-node-log\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.109745 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.109699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-cni-netd\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.109745 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.109715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/350fba86-71d9-4d96-b9ed-de22ab9333c9-konnectivity-ca\") pod \"konnectivity-agent-rpnzb\" (UID: \"350fba86-71d9-4d96-b9ed-de22ab9333c9\") " pod="kube-system/konnectivity-agent-rpnzb"
Apr 16 16:29:52.109863 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.109752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-run-k8s-cni-cncf-io\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.109863 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.109776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c76q2\" (UniqueName: \"kubernetes.io/projected/57674e7d-ee63-4a14-85d1-8c35f765106d-kube-api-access-c76q2\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.109863 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.109810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-sysctl-d\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.109863 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.109837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-var-lib-kubelet\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.109979 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.109862    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-etc-kubernetes\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.109979 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.109896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f98643b4-a68c-4a82-ac48-50ebf25135ba-tmp\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.109979 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.109922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-kubelet\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.109979 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.109939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12c1b394-8723-4774-aecd-3c20c35fc722-hosts-file\") pod \"node-resolver-fbgtp\" (UID: \"12c1b394-8723-4774-aecd-3c20c35fc722\") " pod="openshift-dns/node-resolver-fbgtp"
Apr 16 16:29:52.109979 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.109958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-system-cni-dir\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.110115 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.109981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57674e7d-ee63-4a14-85d1-8c35f765106d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.110115 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcj8b\" (UniqueName: \"kubernetes.io/projected/21a9695d-812c-4ad6-9b18-ade579b343e0-kube-api-access-fcj8b\") pod \"network-metrics-daemon-wq65n\" (UID: \"21a9695d-812c-4ad6-9b18-ade579b343e0\") " pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:29:52.110115 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-lib-modules\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.110115 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-host\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.110115 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-run-netns\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.110115 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-sysctl-conf\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.110115 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57674e7d-ee63-4a14-85d1-8c35f765106d-os-release\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.110312 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d64db671-44fb-4c47-bda6-b8c720247f41-host-slash\") pod \"iptables-alerter-xdwz4\" (UID: \"d64db671-44fb-4c47-bda6-b8c720247f41\") " pod="openshift-network-operator/iptables-alerter-xdwz4"
Apr 16 16:29:52.110312 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-ovnkube-config\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.110312 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57674e7d-ee63-4a14-85d1-8c35f765106d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.110312 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbpcb\" (UniqueName: \"kubernetes.io/projected/d64db671-44fb-4c47-bda6-b8c720247f41-kube-api-access-qbpcb\") pod \"iptables-alerter-xdwz4\" (UID: \"d64db671-44fb-4c47-bda6-b8c720247f41\") " pod="openshift-network-operator/iptables-alerter-xdwz4"
Apr 16 16:29:52.110312 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-sysconfig\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.110312 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-systemd\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.110312 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-ovn-node-metrics-cert\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.110312 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-var-lib-cni-bin\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.110312 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57674e7d-ee63-4a14-85d1-8c35f765106d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.110312 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf5wx\" (UniqueName: \"kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx\") pod \"network-check-target-w2mqw\" (UID: \"29bd11bf-6683-4b59-9b71-e5f556a0ad04\") " pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:29:52.110312 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-sys\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.110312 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-run-ovn\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.110660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110340    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-log-socket\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.110660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57674e7d-ee63-4a14-85d1-8c35f765106d-system-cni-dir\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.110660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/57674e7d-ee63-4a14-85d1-8c35f765106d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.110660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-var-lib-openvswitch\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.110660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-run-ovn-kubernetes\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.110660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-var-lib-cni-multus\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.110660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-hostroot\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.110660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8trdj\" (UniqueName: \"kubernetes.io/projected/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-kube-api-access-8trdj\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.110660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/350fba86-71d9-4d96-b9ed-de22ab9333c9-agent-certs\") pod \"konnectivity-agent-rpnzb\" (UID: \"350fba86-71d9-4d96-b9ed-de22ab9333c9\") " pod="kube-system/konnectivity-agent-rpnzb"
Apr 16 16:29:52.110660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c1b394-8723-4774-aecd-3c20c35fc722-tmp-dir\") pod \"node-resolver-fbgtp\" (UID: \"12c1b394-8723-4774-aecd-3c20c35fc722\") " pod="openshift-dns/node-resolver-fbgtp"
Apr 16 16:29:52.110660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-run-multus-certs\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.110660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-env-overrides\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.110660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5d3493d-e664-4bec-a956-08fc7c9bbf03-cni-binary-copy\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.110660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-run-netns\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.110660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-multus-conf-dir\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.110660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-socket-dir\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqj5d\" (UniqueName: \"kubernetes.io/projected/c5d3493d-e664-4bec-a956-08fc7c9bbf03-kube-api-access-mqj5d\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-kubernetes\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9drrw\" (UniqueName: \"kubernetes.io/projected/f98643b4-a68c-4a82-ac48-50ebf25135ba-kube-api-access-9drrw\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-systemd-units\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-cni-bin\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-multus-cni-dir\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-cnibin\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57674e7d-ee63-4a14-85d1-8c35f765106d-cnibin\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-registration-dir\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-tuned\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110955    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal"]
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110975    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d36c965f-65b5-4340-8151-e2ff609581b6-host\") pod \"node-ca-7qft9\" (UID: \"d36c965f-65b5-4340-8151-e2ff609581b6\") " pod="openshift-image-registry/node-ca-7qft9"
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.110998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-run-openvswitch\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111035    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-os-release\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.111127 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c5d3493d-e664-4bec-a956-08fc7c9bbf03-multus-daemon-config\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.111669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d64db671-44fb-4c47-bda6-b8c720247f41-iptables-alerter-script\") pod \"iptables-alerter-xdwz4\" (UID: \"d64db671-44fb-4c47-bda6-b8c720247f41\") " pod="openshift-network-operator/iptables-alerter-xdwz4"
Apr 16 16:29:52.111669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-device-dir\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.111669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxrbs\" (UniqueName: \"kubernetes.io/projected/d36c965f-65b5-4340-8151-e2ff609581b6-kube-api-access-xxrbs\") pod \"node-ca-7qft9\" (UID: \"d36c965f-65b5-4340-8151-e2ff609581b6\") " pod="openshift-image-registry/node-ca-7qft9"
Apr 16 16:29:52.111669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-run-systemd\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.111669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-ovnkube-script-lib\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.111669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zfqf\" (UniqueName: \"kubernetes.io/projected/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-kube-api-access-8zfqf\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.111669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kc7n\" (UniqueName: \"kubernetes.io/projected/12c1b394-8723-4774-aecd-3c20c35fc722-kube-api-access-4kc7n\") pod \"node-resolver-fbgtp\" (UID: \"12c1b394-8723-4774-aecd-3c20c35fc722\") " pod="openshift-dns/node-resolver-fbgtp"
Apr 16 16:29:52.111669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-etc-selinux\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.111669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-modprobe-d\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.111669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-var-lib-kubelet\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.111669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d36c965f-65b5-4340-8151-e2ff609581b6-serviceca\") pod \"node-ca-7qft9\" (UID: \"d36c965f-65b5-4340-8151-e2ff609581b6\") " pod="openshift-image-registry/node-ca-7qft9"
Apr 16 16:29:52.111669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-etc-openvswitch\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.111669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-multus-socket-dir-parent\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.111669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs\") pod \"network-metrics-daemon-wq65n\" (UID: \"21a9695d-812c-4ad6-9b18-ade579b343e0\") " pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:29:52.111669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-sys-fs\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.112093 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111775    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Apr 16 16:29:52.112093 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.111827    2578 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.ec2.internal"
Apr 16 16:29:52.113336 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.113322    2578 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Apr 16 16:29:52.119344 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.119322    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-164.ec2.internal"]
Apr 16 16:29:52.119423 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.119353    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Apr 16 16:29:52.133418 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.133397    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-k24pg"
Apr 16 16:29:52.140887 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.140867    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k24pg"
Apr 16 16:29:52.172710 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:52.172681    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4ffc4283ad098ce5617bb701ba0817.slice/crio-bd781fc53d0329af22409045dc58e6b84700485601aec0f5a36856054cd99a3e WatchSource:0}: Error finding container bd781fc53d0329af22409045dc58e6b84700485601aec0f5a36856054cd99a3e: Status 404 returned error can't find the container with id bd781fc53d0329af22409045dc58e6b84700485601aec0f5a36856054cd99a3e
Apr 16 16:29:52.173148 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:52.173130    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a26939a85c3397690ad7fddb03cc34.slice/crio-ebf942eb0996e4b6754d91985259afb04f4e94715e08cce43e60eb816fd589e1 WatchSource:0}: Error finding container ebf942eb0996e4b6754d91985259afb04f4e94715e08cce43e60eb816fd589e1: Status 404 returned error can't find the container with id ebf942eb0996e4b6754d91985259afb04f4e94715e08cce43e60eb816fd589e1
Apr 16 16:29:52.179343 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.179328    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 16:29:52.211842 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.211824    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-kubernetes\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.211952 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.211848    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9drrw\" (UniqueName: \"kubernetes.io/projected/f98643b4-a68c-4a82-ac48-50ebf25135ba-kube-api-access-9drrw\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.211952 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.211864    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-systemd-units\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.211952 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.211878    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-cni-bin\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.211952 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.211893    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-multus-cni-dir\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.211952 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.211947    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-kubernetes\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.211954    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-multus-cni-dir\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.211948    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-cni-bin\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.211969    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-cnibin\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.211994    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-systemd-units\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.211995    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57674e7d-ee63-4a14-85d1-8c35f765106d-cnibin\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212020    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-registration-dir\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212025    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57674e7d-ee63-4a14-85d1-8c35f765106d-cnibin\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212037    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-tuned\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212060    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d36c965f-65b5-4340-8151-e2ff609581b6-host\") pod \"node-ca-7qft9\" (UID: \"d36c965f-65b5-4340-8151-e2ff609581b6\") " pod="openshift-image-registry/node-ca-7qft9"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212084    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-run-openvswitch\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212088    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-cnibin\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212097    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-registration-dir\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212123    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-os-release\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212139    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d36c965f-65b5-4340-8151-e2ff609581b6-host\") pod \"node-ca-7qft9\" (UID: \"d36c965f-65b5-4340-8151-e2ff609581b6\") " pod="openshift-image-registry/node-ca-7qft9"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212148    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c5d3493d-e664-4bec-a956-08fc7c9bbf03-multus-daemon-config\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212151    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-run-openvswitch\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212170    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d64db671-44fb-4c47-bda6-b8c720247f41-iptables-alerter-script\") pod \"iptables-alerter-xdwz4\" (UID: \"d64db671-44fb-4c47-bda6-b8c720247f41\") " pod="openshift-network-operator/iptables-alerter-xdwz4"
Apr 16 16:29:52.212207 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212190    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-os-release\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212195    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-device-dir\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212221    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xxrbs\" (UniqueName: \"kubernetes.io/projected/d36c965f-65b5-4340-8151-e2ff609581b6-kube-api-access-xxrbs\") pod \"node-ca-7qft9\" (UID: \"d36c965f-65b5-4340-8151-e2ff609581b6\") " pod="openshift-image-registry/node-ca-7qft9"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212244    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-run-systemd\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212268    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-ovnkube-script-lib\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212276    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-device-dir\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212301    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-run-systemd\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212345    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212372    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfqf\" (UniqueName: \"kubernetes.io/projected/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-kube-api-access-8zfqf\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212394    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4kc7n\" (UniqueName: \"kubernetes.io/projected/12c1b394-8723-4774-aecd-3c20c35fc722-kube-api-access-4kc7n\") pod \"node-resolver-fbgtp\" (UID: \"12c1b394-8723-4774-aecd-3c20c35fc722\") " pod="openshift-dns/node-resolver-fbgtp"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212412    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-etc-selinux\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212427    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-modprobe-d\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212448    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-var-lib-kubelet\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212471    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d36c965f-65b5-4340-8151-e2ff609581b6-serviceca\") pod \"node-ca-7qft9\" (UID: \"d36c965f-65b5-4340-8151-e2ff609581b6\") " pod="openshift-image-registry/node-ca-7qft9"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212494    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-etc-openvswitch\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212513    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-etc-selinux\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212621    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-modprobe-d\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212640    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-multus-socket-dir-parent\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.213084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212516    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-multus-socket-dir-parent\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212695    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs\") pod \"network-metrics-daemon-wq65n\" (UID: \"21a9695d-812c-4ad6-9b18-ade579b343e0\") " pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212725    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-sys-fs\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212750    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-run\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212783    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d64db671-44fb-4c47-bda6-b8c720247f41-iptables-alerter-script\") pod \"iptables-alerter-xdwz4\" (UID: \"d64db671-44fb-4c47-bda6-b8c720247f41\") " pod="openshift-network-operator/iptables-alerter-xdwz4"
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212791    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-slash\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212796    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-etc-openvswitch\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212816    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-node-log\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212695    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-var-lib-kubelet\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:52.212840    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212854    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-run\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212869    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-node-log\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212879    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-cni-netd\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212878    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-slash\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212928    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-sys-fs\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212937    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-cni-netd\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212937    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-ovnkube-script-lib\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.213846 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:52.212952    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs podName:21a9695d-812c-4ad6-9b18-ade579b343e0 nodeName:}" failed. No retries permitted until 2026-04-16 16:29:52.712889937 +0000 UTC m=+2.110106331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs") pod "network-metrics-daemon-wq65n" (UID: "21a9695d-812c-4ad6-9b18-ade579b343e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.212977    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/350fba86-71d9-4d96-b9ed-de22ab9333c9-konnectivity-ca\") pod \"konnectivity-agent-rpnzb\" (UID: \"350fba86-71d9-4d96-b9ed-de22ab9333c9\") " pod="kube-system/konnectivity-agent-rpnzb"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213005    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-run-k8s-cni-cncf-io\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213020    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c5d3493d-e664-4bec-a956-08fc7c9bbf03-multus-daemon-config\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213031    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c76q2\" (UniqueName: \"kubernetes.io/projected/57674e7d-ee63-4a14-85d1-8c35f765106d-kube-api-access-c76q2\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213059    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-sysctl-d\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213075    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-run-k8s-cni-cncf-io\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213083    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-var-lib-kubelet\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213081    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d36c965f-65b5-4340-8151-e2ff609581b6-serviceca\") pod \"node-ca-7qft9\" (UID: \"d36c965f-65b5-4340-8151-e2ff609581b6\") " pod="openshift-image-registry/node-ca-7qft9"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213122    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-etc-kubernetes\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213149    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f98643b4-a68c-4a82-ac48-50ebf25135ba-tmp\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213163    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-etc-kubernetes\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213174    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-kubelet\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213125    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-var-lib-kubelet\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213193    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-sysctl-d\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213199    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12c1b394-8723-4774-aecd-3c20c35fc722-hosts-file\") pod \"node-resolver-fbgtp\" (UID: \"12c1b394-8723-4774-aecd-3c20c35fc722\") " pod="openshift-dns/node-resolver-fbgtp"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213229    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-system-cni-dir\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213239    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-kubelet\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.214572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213244    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12c1b394-8723-4774-aecd-3c20c35fc722-hosts-file\") pod \"node-resolver-fbgtp\" (UID: \"12c1b394-8723-4774-aecd-3c20c35fc722\") " pod="openshift-dns/node-resolver-fbgtp"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213256    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57674e7d-ee63-4a14-85d1-8c35f765106d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213276    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fcj8b\" (UniqueName: \"kubernetes.io/projected/21a9695d-812c-4ad6-9b18-ade579b343e0-kube-api-access-fcj8b\") pod \"network-metrics-daemon-wq65n\" (UID: \"21a9695d-812c-4ad6-9b18-ade579b343e0\") " pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213290    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-system-cni-dir\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213747    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/350fba86-71d9-4d96-b9ed-de22ab9333c9-konnectivity-ca\") pod \"konnectivity-agent-rpnzb\" (UID: \"350fba86-71d9-4d96-b9ed-de22ab9333c9\") " pod="kube-system/konnectivity-agent-rpnzb"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213900    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-lib-modules\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213946    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-host\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.213987    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-run-netns\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214018    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-sysctl-conf\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214045    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57674e7d-ee63-4a14-85d1-8c35f765106d-os-release\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214053    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-lib-modules\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214078    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d64db671-44fb-4c47-bda6-b8c720247f41-host-slash\") pod \"iptables-alerter-xdwz4\" (UID: \"d64db671-44fb-4c47-bda6-b8c720247f41\") " pod="openshift-network-operator/iptables-alerter-xdwz4"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214097    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-host\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214109    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-ovnkube-config\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214129    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57674e7d-ee63-4a14-85d1-8c35f765106d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214157    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57674e7d-ee63-4a14-85d1-8c35f765106d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214188    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qbpcb\" (UniqueName: \"kubernetes.io/projected/d64db671-44fb-4c47-bda6-b8c720247f41-kube-api-access-qbpcb\") pod \"iptables-alerter-xdwz4\" (UID: \"d64db671-44fb-4c47-bda6-b8c720247f41\") " pod="openshift-network-operator/iptables-alerter-xdwz4"
Apr 16 16:29:52.215307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214220    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57674e7d-ee63-4a14-85d1-8c35f765106d-os-release\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214237    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-sysconfig\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214271    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-run-netns\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214273    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-systemd\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214317    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-ovn-node-metrics-cert\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214345    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-systemd\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214346    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-var-lib-cni-bin\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214387    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57674e7d-ee63-4a14-85d1-8c35f765106d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214393    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-var-lib-cni-bin\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214423    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5wx\" (UniqueName: \"kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx\") pod \"network-check-target-w2mqw\" (UID: \"29bd11bf-6683-4b59-9b71-e5f556a0ad04\") " pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214456    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-sys\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214485    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-run-ovn\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214492    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-sysctl-conf\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214516    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-log-socket\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214545    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57674e7d-ee63-4a14-85d1-8c35f765106d-system-cni-dir\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214601    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/57674e7d-ee63-4a14-85d1-8c35f765106d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214610    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-ovnkube-config\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.216049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214636    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-var-lib-openvswitch\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214677    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-run-ovn-kubernetes\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214723    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-var-lib-cni-multus\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214731    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-run-ovn-kubernetes\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214760    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-hostroot\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214794    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8trdj\" (UniqueName: \"kubernetes.io/projected/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-kube-api-access-8trdj\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214820    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/350fba86-71d9-4d96-b9ed-de22ab9333c9-agent-certs\") pod \"konnectivity-agent-rpnzb\" (UID: \"350fba86-71d9-4d96-b9ed-de22ab9333c9\") " pod="kube-system/konnectivity-agent-rpnzb"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214850    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c1b394-8723-4774-aecd-3c20c35fc722-tmp-dir\") pod \"node-resolver-fbgtp\" (UID: \"12c1b394-8723-4774-aecd-3c20c35fc722\") " pod="openshift-dns/node-resolver-fbgtp"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214880    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-run-multus-certs\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214912    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-env-overrides\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214937    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-sys\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214964    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-run-ovn\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.214980    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d64db671-44fb-4c47-bda6-b8c720247f41-host-slash\") pod \"iptables-alerter-xdwz4\" (UID: \"d64db671-44fb-4c47-bda6-b8c720247f41\") " pod="openshift-network-operator/iptables-alerter-xdwz4"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215016    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-var-lib-cni-multus\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215030    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-log-socket\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215078    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-hostroot\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215085    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57674e7d-ee63-4a14-85d1-8c35f765106d-system-cni-dir\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215273    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-env-overrides\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.216761 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215363    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-var-lib-openvswitch\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215368    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c1b394-8723-4774-aecd-3c20c35fc722-tmp-dir\") pod \"node-resolver-fbgtp\" (UID: \"12c1b394-8723-4774-aecd-3c20c35fc722\") " pod="openshift-dns/node-resolver-fbgtp"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215411    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-run-multus-certs\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215461    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5d3493d-e664-4bec-a956-08fc7c9bbf03-cni-binary-copy\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215498    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-run-netns\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215567    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-host-run-netns\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215696    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-tuned\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215767    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-multus-conf-dir\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215830    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215864    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-socket-dir\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215899    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.215939    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mqj5d\" (UniqueName: \"kubernetes.io/projected/c5d3493d-e664-4bec-a956-08fc7c9bbf03-kube-api-access-mqj5d\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.216121    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57674e7d-ee63-4a14-85d1-8c35f765106d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.216170    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/57674e7d-ee63-4a14-85d1-8c35f765106d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.216250    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f98643b4-a68c-4a82-ac48-50ebf25135ba-etc-sysconfig\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.216272    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5d3493d-e664-4bec-a956-08fc7c9bbf03-multus-conf-dir\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.216335    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.217517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.216330    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-socket-dir\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.218322 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.216407    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.218322 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.216509    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5d3493d-e664-4bec-a956-08fc7c9bbf03-cni-binary-copy\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.218322 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.216665    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57674e7d-ee63-4a14-85d1-8c35f765106d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.218322 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.218046    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-ovn-node-metrics-cert\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.218322 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.218158    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f98643b4-a68c-4a82-ac48-50ebf25135ba-tmp\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.219694 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.219673    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/350fba86-71d9-4d96-b9ed-de22ab9333c9-agent-certs\") pod \"konnectivity-agent-rpnzb\" (UID: \"350fba86-71d9-4d96-b9ed-de22ab9333c9\") " pod="kube-system/konnectivity-agent-rpnzb"
Apr 16 16:29:52.219917 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.219900    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kc7n\" (UniqueName: \"kubernetes.io/projected/12c1b394-8723-4774-aecd-3c20c35fc722-kube-api-access-4kc7n\") pod \"node-resolver-fbgtp\" (UID: \"12c1b394-8723-4774-aecd-3c20c35fc722\") " pod="openshift-dns/node-resolver-fbgtp"
Apr 16 16:29:52.219953 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.219914    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9drrw\" (UniqueName: \"kubernetes.io/projected/f98643b4-a68c-4a82-ac48-50ebf25135ba-kube-api-access-9drrw\") pod \"tuned-pdmjc\" (UID: \"f98643b4-a68c-4a82-ac48-50ebf25135ba\") " pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.226061 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:52.226037    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 16 16:29:52.226061 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:52.226063    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 16 16:29:52.226247 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:52.226077    2578 projected.go:194] Error preparing data for projected volume kube-api-access-zf5wx for pod openshift-network-diagnostics/network-check-target-w2mqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:29:52.226247 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:52.226126    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx podName:29bd11bf-6683-4b59-9b71-e5f556a0ad04 nodeName:}" failed. No retries permitted until 2026-04-16 16:29:52.726110004 +0000 UTC m=+2.123326397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zf5wx" (UniqueName: "kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx") pod "network-check-target-w2mqw" (UID: "29bd11bf-6683-4b59-9b71-e5f556a0ad04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:29:52.227234 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.227210    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxrbs\" (UniqueName: \"kubernetes.io/projected/d36c965f-65b5-4340-8151-e2ff609581b6-kube-api-access-xxrbs\") pod \"node-ca-7qft9\" (UID: \"d36c965f-65b5-4340-8151-e2ff609581b6\") " pod="openshift-image-registry/node-ca-7qft9"
Apr 16 16:29:52.227849 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.227825    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c76q2\" (UniqueName: \"kubernetes.io/projected/57674e7d-ee63-4a14-85d1-8c35f765106d-kube-api-access-c76q2\") pod \"multus-additional-cni-plugins-jnptn\" (UID: \"57674e7d-ee63-4a14-85d1-8c35f765106d\") " pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.228423 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.228403    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zfqf\" (UniqueName: \"kubernetes.io/projected/78b0d7e4-bab7-4f8d-ac1e-f45bede95414-kube-api-access-8zfqf\") pod \"ovnkube-node-wd8ld\" (UID: \"78b0d7e4-bab7-4f8d-ac1e-f45bede95414\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.228763 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.228745    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcj8b\" (UniqueName: \"kubernetes.io/projected/21a9695d-812c-4ad6-9b18-ade579b343e0-kube-api-access-fcj8b\") pod \"network-metrics-daemon-wq65n\" (UID: \"21a9695d-812c-4ad6-9b18-ade579b343e0\") " pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:29:52.228836 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.228761    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqj5d\" (UniqueName: \"kubernetes.io/projected/c5d3493d-e664-4bec-a956-08fc7c9bbf03-kube-api-access-mqj5d\") pod \"multus-7cqz2\" (UID: \"c5d3493d-e664-4bec-a956-08fc7c9bbf03\") " pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.228980 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.228964    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8trdj\" (UniqueName: \"kubernetes.io/projected/158e69b7-65d9-4169-9fdb-cdb6e74eecb6-kube-api-access-8trdj\") pod \"aws-ebs-csi-driver-node-zl6ns\" (UID: \"158e69b7-65d9-4169-9fdb-cdb6e74eecb6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.229649 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.229633    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbpcb\" (UniqueName: \"kubernetes.io/projected/d64db671-44fb-4c47-bda6-b8c720247f41-kube-api-access-qbpcb\") pod \"iptables-alerter-xdwz4\" (UID: \"d64db671-44fb-4c47-bda6-b8c720247f41\") " pod="openshift-network-operator/iptables-alerter-xdwz4"
Apr 16 16:29:52.233182 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.233148    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.ec2.internal" event={"ID":"0a4ffc4283ad098ce5617bb701ba0817","Type":"ContainerStarted","Data":"bd781fc53d0329af22409045dc58e6b84700485601aec0f5a36856054cd99a3e"}
Apr 16 16:29:52.233982 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.233965    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal" event={"ID":"c3a26939a85c3397690ad7fddb03cc34","Type":"ContainerStarted","Data":"ebf942eb0996e4b6754d91985259afb04f4e94715e08cce43e60eb816fd589e1"}
Apr 16 16:29:52.419932 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.419871    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fbgtp"
Apr 16 16:29:52.425167 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.425149    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7cqz2"
Apr 16 16:29:52.425966 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:52.425941    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c1b394_8723_4774_aecd_3c20c35fc722.slice/crio-67cb1154f0d5e23562038022609cacf72ebc14bfedc8f9c7dd45957668ebcf8e WatchSource:0}: Error finding container 67cb1154f0d5e23562038022609cacf72ebc14bfedc8f9c7dd45957668ebcf8e: Status 404 returned error can't find the container with id 67cb1154f0d5e23562038022609cacf72ebc14bfedc8f9c7dd45957668ebcf8e
Apr 16 16:29:52.431382 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:52.431355    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d3493d_e664_4bec_a956_08fc7c9bbf03.slice/crio-12a77ce98fa31a9f7865e5ce8d54a2a360c9f81af3cac81cf5fb9da03c5b6cd6 WatchSource:0}: Error finding container 12a77ce98fa31a9f7865e5ce8d54a2a360c9f81af3cac81cf5fb9da03c5b6cd6: Status 404 returned error can't find the container with id 12a77ce98fa31a9f7865e5ce8d54a2a360c9f81af3cac81cf5fb9da03c5b6cd6
Apr 16 16:29:52.451841 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.451811    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jnptn"
Apr 16 16:29:52.457747 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:52.457724    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57674e7d_ee63_4a14_85d1_8c35f765106d.slice/crio-5484c9a7be2d61b7d1497dfb9fd912d91afc8a9908c37b60e41e581b85c4eae2 WatchSource:0}: Error finding container 5484c9a7be2d61b7d1497dfb9fd912d91afc8a9908c37b60e41e581b85c4eae2: Status 404 returned error can't find the container with id 5484c9a7be2d61b7d1497dfb9fd912d91afc8a9908c37b60e41e581b85c4eae2
Apr 16 16:29:52.473126 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.473091    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xdwz4"
Apr 16 16:29:52.478398 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:52.478376    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64db671_44fb_4c47_bda6_b8c720247f41.slice/crio-560a0e310efd22973cd610399c088f8b7523f0f5e149d299b278421aa73bd05e WatchSource:0}: Error finding container 560a0e310efd22973cd610399c088f8b7523f0f5e149d299b278421aa73bd05e: Status 404 returned error can't find the container with id 560a0e310efd22973cd610399c088f8b7523f0f5e149d299b278421aa73bd05e
Apr 16 16:29:52.479047 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.479031    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns"
Apr 16 16:29:52.484495 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:52.484475    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158e69b7_65d9_4169_9fdb_cdb6e74eecb6.slice/crio-75ba4e1da6c232b1aa8ed32dc38478ade845dd805a866b2e9fd46f6172da4098 WatchSource:0}: Error finding container 75ba4e1da6c232b1aa8ed32dc38478ade845dd805a866b2e9fd46f6172da4098: Status 404 returned error can't find the container with id 75ba4e1da6c232b1aa8ed32dc38478ade845dd805a866b2e9fd46f6172da4098
Apr 16 16:29:52.485055 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.485041    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pdmjc"
Apr 16 16:29:52.491037 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.491015    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7qft9"
Apr 16 16:29:52.491169 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:52.491142    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98643b4_a68c_4a82_ac48_50ebf25135ba.slice/crio-fa4a8a2652d08099d87c77c191b09d1b8f933aad0efcd1ed54a9057258dfe646 WatchSource:0}: Error finding container fa4a8a2652d08099d87c77c191b09d1b8f933aad0efcd1ed54a9057258dfe646: Status 404 returned error can't find the container with id fa4a8a2652d08099d87c77c191b09d1b8f933aad0efcd1ed54a9057258dfe646
Apr 16 16:29:52.492499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.492485    2578 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 16 16:29:52.496660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.496600    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:29:52.496774 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:52.496729    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36c965f_65b5_4340_8151_e2ff609581b6.slice/crio-6f114b853b918610f00ffa9a2588e0aa0146ba7e680e66ecf1dc76bfc5684ff5 WatchSource:0}: Error finding container 6f114b853b918610f00ffa9a2588e0aa0146ba7e680e66ecf1dc76bfc5684ff5: Status 404 returned error can't find the container with id 6f114b853b918610f00ffa9a2588e0aa0146ba7e680e66ecf1dc76bfc5684ff5
Apr 16 16:29:52.501041 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.500991    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rpnzb"
Apr 16 16:29:52.504807 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:52.504786    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b0d7e4_bab7_4f8d_ac1e_f45bede95414.slice/crio-e8e7dfe3bcf32159c72d458442b6c031db063302f97aeec68cf6f172f3eb6a38 WatchSource:0}: Error finding container e8e7dfe3bcf32159c72d458442b6c031db063302f97aeec68cf6f172f3eb6a38: Status 404 returned error can't find the container with id e8e7dfe3bcf32159c72d458442b6c031db063302f97aeec68cf6f172f3eb6a38
Apr 16 16:29:52.509222 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:29:52.509198    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350fba86_71d9_4d96_b9ed_de22ab9333c9.slice/crio-0ea8a43d40f35dc784d9ff7921523569a8b0c57cef941c22807d4ff8b5be3fbf WatchSource:0}: Error finding container 0ea8a43d40f35dc784d9ff7921523569a8b0c57cef941c22807d4ff8b5be3fbf: Status 404 returned error can't find the container with id 0ea8a43d40f35dc784d9ff7921523569a8b0c57cef941c22807d4ff8b5be3fbf
Apr 16 16:29:52.719263 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.719184    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs\") pod \"network-metrics-daemon-wq65n\" (UID: \"21a9695d-812c-4ad6-9b18-ade579b343e0\") " pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:29:52.719412 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:52.719342    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:29:52.719412 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:52.719409    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs podName:21a9695d-812c-4ad6-9b18-ade579b343e0 nodeName:}" failed. No retries permitted until 2026-04-16 16:29:53.719387852 +0000 UTC m=+3.116604231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs") pod "network-metrics-daemon-wq65n" (UID: "21a9695d-812c-4ad6-9b18-ade579b343e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:29:52.820974 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:52.820382    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5wx\" (UniqueName: \"kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx\") pod \"network-check-target-w2mqw\" (UID: \"29bd11bf-6683-4b59-9b71-e5f556a0ad04\") " pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:29:52.820974 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:52.820527    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 16 16:29:52.820974 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:52.820547    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 16 16:29:52.820974 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:52.820576    2578 projected.go:194] Error preparing data for projected volume kube-api-access-zf5wx for pod openshift-network-diagnostics/network-check-target-w2mqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:29:52.820974 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:52.820632    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx podName:29bd11bf-6683-4b59-9b71-e5f556a0ad04 nodeName:}" failed. No retries permitted until 2026-04-16 16:29:53.820612695 +0000 UTC m=+3.217829064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf5wx" (UniqueName: "kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx") pod "network-check-target-w2mqw" (UID: "29bd11bf-6683-4b59-9b71-e5f556a0ad04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:29:53.141659 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:53.141508    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-15 16:24:52 +0000 UTC" deadline="2027-11-27 20:30:50.973924114 +0000 UTC"
Apr 16 16:29:53.141659 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:53.141549    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14164h0m57.832379896s"
Apr 16 16:29:53.209580 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:53.209370    2578 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 16 16:29:53.248614 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:53.248528    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rpnzb" event={"ID":"350fba86-71d9-4d96-b9ed-de22ab9333c9","Type":"ContainerStarted","Data":"0ea8a43d40f35dc784d9ff7921523569a8b0c57cef941c22807d4ff8b5be3fbf"}
Apr 16 16:29:53.268612 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:53.268545    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns" event={"ID":"158e69b7-65d9-4169-9fdb-cdb6e74eecb6","Type":"ContainerStarted","Data":"75ba4e1da6c232b1aa8ed32dc38478ade845dd805a866b2e9fd46f6172da4098"}
Apr 16 16:29:53.285498 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:53.285455    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld" event={"ID":"78b0d7e4-bab7-4f8d-ac1e-f45bede95414","Type":"ContainerStarted","Data":"e8e7dfe3bcf32159c72d458442b6c031db063302f97aeec68cf6f172f3eb6a38"}
Apr 16 16:29:53.288944 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:53.288904    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7qft9" event={"ID":"d36c965f-65b5-4340-8151-e2ff609581b6","Type":"ContainerStarted","Data":"6f114b853b918610f00ffa9a2588e0aa0146ba7e680e66ecf1dc76bfc5684ff5"}
Apr 16 16:29:53.290980 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:53.290914    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pdmjc" event={"ID":"f98643b4-a68c-4a82-ac48-50ebf25135ba","Type":"ContainerStarted","Data":"fa4a8a2652d08099d87c77c191b09d1b8f933aad0efcd1ed54a9057258dfe646"}
Apr 16 16:29:53.308051 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:53.308023    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xdwz4" event={"ID":"d64db671-44fb-4c47-bda6-b8c720247f41","Type":"ContainerStarted","Data":"560a0e310efd22973cd610399c088f8b7523f0f5e149d299b278421aa73bd05e"}
Apr 16 16:29:53.315775 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:53.315748    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnptn" event={"ID":"57674e7d-ee63-4a14-85d1-8c35f765106d","Type":"ContainerStarted","Data":"5484c9a7be2d61b7d1497dfb9fd912d91afc8a9908c37b60e41e581b85c4eae2"}
Apr 16 16:29:53.334221 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:53.334146    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7cqz2" event={"ID":"c5d3493d-e664-4bec-a956-08fc7c9bbf03","Type":"ContainerStarted","Data":"12a77ce98fa31a9f7865e5ce8d54a2a360c9f81af3cac81cf5fb9da03c5b6cd6"}
Apr 16 16:29:53.340427 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:53.340353    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fbgtp" event={"ID":"12c1b394-8723-4774-aecd-3c20c35fc722","Type":"ContainerStarted","Data":"67cb1154f0d5e23562038022609cacf72ebc14bfedc8f9c7dd45957668ebcf8e"}
Apr 16 16:29:53.367919 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:53.367846    2578 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 16 16:29:53.729318 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:53.728591    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs\") pod \"network-metrics-daemon-wq65n\" (UID: \"21a9695d-812c-4ad6-9b18-ade579b343e0\") " pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:29:53.729318 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:53.728754    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:29:53.729318 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:53.728818    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs podName:21a9695d-812c-4ad6-9b18-ade579b343e0 nodeName:}" failed. No retries permitted until 2026-04-16 16:29:55.728799522 +0000 UTC m=+5.126015906 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs") pod "network-metrics-daemon-wq65n" (UID: "21a9695d-812c-4ad6-9b18-ade579b343e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:29:53.829524 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:53.829447    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5wx\" (UniqueName: \"kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx\") pod \"network-check-target-w2mqw\" (UID: \"29bd11bf-6683-4b59-9b71-e5f556a0ad04\") " pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:29:53.829722 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:53.829624    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 16 16:29:53.829722 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:53.829648    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 16 16:29:53.829722 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:53.829661    2578 projected.go:194] Error preparing data for projected volume kube-api-access-zf5wx for pod openshift-network-diagnostics/network-check-target-w2mqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:29:53.829722 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:53.829716    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx podName:29bd11bf-6683-4b59-9b71-e5f556a0ad04 nodeName:}" failed. No retries permitted until 2026-04-16 16:29:55.829698119 +0000 UTC m=+5.226914489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf5wx" (UniqueName: "kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx") pod "network-check-target-w2mqw" (UID: "29bd11bf-6683-4b59-9b71-e5f556a0ad04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:29:54.142309 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:54.142197    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-15 16:24:52 +0000 UTC" deadline="2027-09-25 22:35:35.053950739 +0000 UTC"
Apr 16 16:29:54.142309 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:54.142232    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12654h5m40.911722506s"
Apr 16 16:29:54.232375 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:54.231446    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:29:54.232375 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:54.231582    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:29:54.232375 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:54.231996    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:29:54.232375 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:54.232096    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:29:55.745772 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:55.745500    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs\") pod \"network-metrics-daemon-wq65n\" (UID: \"21a9695d-812c-4ad6-9b18-ade579b343e0\") " pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:29:55.746233 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:55.745929    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:29:55.746233 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:55.745995    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs podName:21a9695d-812c-4ad6-9b18-ade579b343e0 nodeName:}" failed. No retries permitted until 2026-04-16 16:29:59.745975621 +0000 UTC m=+9.143191989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs") pod "network-metrics-daemon-wq65n" (UID: "21a9695d-812c-4ad6-9b18-ade579b343e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:29:55.847061 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:55.846461    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5wx\" (UniqueName: \"kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx\") pod \"network-check-target-w2mqw\" (UID: \"29bd11bf-6683-4b59-9b71-e5f556a0ad04\") " pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:29:55.847061 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:55.846639    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 16 16:29:55.847061 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:55.846658    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 16 16:29:55.847061 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:55.846670    2578 projected.go:194] Error preparing data for projected volume kube-api-access-zf5wx for pod openshift-network-diagnostics/network-check-target-w2mqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:29:55.847061 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:55.846727    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx podName:29bd11bf-6683-4b59-9b71-e5f556a0ad04 nodeName:}" failed. No retries permitted until 2026-04-16 16:29:59.846708675 +0000 UTC m=+9.243925041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf5wx" (UniqueName: "kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx") pod "network-check-target-w2mqw" (UID: "29bd11bf-6683-4b59-9b71-e5f556a0ad04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:29:56.230934 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:56.230856    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:29:56.230934 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:56.230908    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:29:56.231153 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:56.230979    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:29:56.231289 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:56.231189    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:29:58.230832 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:58.230794    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:29:58.231301 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:58.230806    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:29:58.231301 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:58.230932    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:29:58.231301 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:58.231002    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:29:59.773999 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:59.773965    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs\") pod \"network-metrics-daemon-wq65n\" (UID: \"21a9695d-812c-4ad6-9b18-ade579b343e0\") " pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:29:59.774451 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:59.774120    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:29:59.774451 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:59.774192    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs podName:21a9695d-812c-4ad6-9b18-ade579b343e0 nodeName:}" failed. No retries permitted until 2026-04-16 16:30:07.774172286 +0000 UTC m=+17.171388668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs") pod "network-metrics-daemon-wq65n" (UID: "21a9695d-812c-4ad6-9b18-ade579b343e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:29:59.875594 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:29:59.874991    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5wx\" (UniqueName: \"kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx\") pod \"network-check-target-w2mqw\" (UID: \"29bd11bf-6683-4b59-9b71-e5f556a0ad04\") " pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:29:59.875594 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:59.875173    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 16 16:29:59.875594 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:59.875192    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 16 16:29:59.875594 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:59.875204    2578 projected.go:194] Error preparing data for projected volume kube-api-access-zf5wx for pod openshift-network-diagnostics/network-check-target-w2mqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:29:59.875594 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:29:59.875257    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx podName:29bd11bf-6683-4b59-9b71-e5f556a0ad04 nodeName:}" failed. No retries permitted until 2026-04-16 16:30:07.875239766 +0000 UTC m=+17.272456132 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf5wx" (UniqueName: "kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx") pod "network-check-target-w2mqw" (UID: "29bd11bf-6683-4b59-9b71-e5f556a0ad04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:30:00.230792 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:00.230718    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:00.230940 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:00.230836    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:30:00.230940 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:00.230875    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:00.231059 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:00.230961    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:30:02.231294 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:02.231263    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:02.231742 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:02.231260    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:02.231742 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:02.231393    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:30:02.231742 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:02.231474    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:30:04.230783 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:04.230750    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:04.231197 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:04.230749    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:04.231197 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:04.230891    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:30:04.231197 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:04.230942    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:30:06.231369 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:06.231302    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:06.231759 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:06.231302    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:06.231759 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:06.231393    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:30:06.231759 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:06.231519    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:30:07.835749 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:07.835699    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs\") pod \"network-metrics-daemon-wq65n\" (UID: \"21a9695d-812c-4ad6-9b18-ade579b343e0\") " pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:07.836259 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:07.835825    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:30:07.836259 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:07.835881    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs podName:21a9695d-812c-4ad6-9b18-ade579b343e0 nodeName:}" failed. No retries permitted until 2026-04-16 16:30:23.835861899 +0000 UTC m=+33.233078279 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs") pod "network-metrics-daemon-wq65n" (UID: "21a9695d-812c-4ad6-9b18-ade579b343e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:30:07.936275 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:07.936242    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5wx\" (UniqueName: \"kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx\") pod \"network-check-target-w2mqw\" (UID: \"29bd11bf-6683-4b59-9b71-e5f556a0ad04\") " pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:07.936444 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:07.936384    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 16 16:30:07.936444 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:07.936405    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 16 16:30:07.936444 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:07.936417    2578 projected.go:194] Error preparing data for projected volume kube-api-access-zf5wx for pod openshift-network-diagnostics/network-check-target-w2mqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:30:07.936613 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:07.936474    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx podName:29bd11bf-6683-4b59-9b71-e5f556a0ad04 nodeName:}" failed. No retries permitted until 2026-04-16 16:30:23.936456668 +0000 UTC m=+33.333673041 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf5wx" (UniqueName: "kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx") pod "network-check-target-w2mqw" (UID: "29bd11bf-6683-4b59-9b71-e5f556a0ad04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:30:08.231352 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:08.231328    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:08.231533 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:08.231325    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:08.231533 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:08.231454    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:30:08.231533 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:08.231501    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:30:10.231395 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:10.231274    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:10.231972 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:10.231298    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:10.231972 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:10.231500    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:30:10.231972 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:10.231616    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:30:10.381420 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:10.381383    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7cqz2" event={"ID":"c5d3493d-e664-4bec-a956-08fc7c9bbf03","Type":"ContainerStarted","Data":"7711e5edce95e49524b99204786238025ffee7187a7985adcbd725910f66a456"}
Apr 16 16:30:10.385290 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:10.385226    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.ec2.internal" event={"ID":"0a4ffc4283ad098ce5617bb701ba0817","Type":"ContainerStarted","Data":"3112abf15c18895b2dd17372a7002f6845236b22ad051b86d4cb1c3246eec285"}
Apr 16 16:30:10.389330 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:10.389303    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 16:30:10.390191 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:10.390029    2578 generic.go:358] "Generic (PLEG): container finished" podID="78b0d7e4-bab7-4f8d-ac1e-f45bede95414" containerID="0f15a80e9f8fd4977016c3686f6427d572d293c4ba61c269f5ca9e710d235dcf" exitCode=1
Apr 16 16:30:10.390191 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:10.390107    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld" event={"ID":"78b0d7e4-bab7-4f8d-ac1e-f45bede95414","Type":"ContainerStarted","Data":"ff59958dc011fe0673b35e46a3d712f9cc54af79a5e08a9707cfc3d0324e1fed"}
Apr 16 16:30:10.390191 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:10.390129    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld" event={"ID":"78b0d7e4-bab7-4f8d-ac1e-f45bede95414","Type":"ContainerStarted","Data":"661c20ef42cf9040613f000ebd79bd199eb543d2808c0efbe5ee2140fc45dbae"}
Apr 16 16:30:10.390191 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:10.390144    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld" event={"ID":"78b0d7e4-bab7-4f8d-ac1e-f45bede95414","Type":"ContainerStarted","Data":"53883997fb0c9fd235a0554178632efa7bc2ceb80c4da7247ea98d18d2b2c778"}
Apr 16 16:30:10.390191 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:10.390156    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld" event={"ID":"78b0d7e4-bab7-4f8d-ac1e-f45bede95414","Type":"ContainerStarted","Data":"5481a0692b41e3b3ba2d8997be1e24ba6048a9ee091cf228624f8f2fbf9ef6dd"}
Apr 16 16:30:10.390191 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:10.390168    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld" event={"ID":"78b0d7e4-bab7-4f8d-ac1e-f45bede95414","Type":"ContainerDied","Data":"0f15a80e9f8fd4977016c3686f6427d572d293c4ba61c269f5ca9e710d235dcf"}
Apr 16 16:30:10.390191 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:10.390183    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld" event={"ID":"78b0d7e4-bab7-4f8d-ac1e-f45bede95414","Type":"ContainerStarted","Data":"09dab67e5e9b0a60a07c978dd40190fdef1aab0c6057a7421b23f8f35b64e2c0"}
Apr 16 16:30:10.391900 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:10.391877    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pdmjc" event={"ID":"f98643b4-a68c-4a82-ac48-50ebf25135ba","Type":"ContainerStarted","Data":"5a51ddd41a21a623a1cae4c17a2e216646ff7497b1eaf5fb36404e9822964bdd"}
Apr 16 16:30:10.398658 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:10.398594    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-7cqz2" podStartSLOduration=1.999569044 podStartE2EDuration="19.398537944s" podCreationTimestamp="2026-04-16 16:29:51 +0000 UTC" firstStartedPulling="2026-04-16 16:29:52.432910807 +0000 UTC m=+1.830127174" lastFinishedPulling="2026-04-16 16:30:09.831879707 +0000 UTC m=+19.229096074" observedRunningTime="2026-04-16 16:30:10.397431255 +0000 UTC m=+19.794647670" watchObservedRunningTime="2026-04-16 16:30:10.398537944 +0000 UTC m=+19.795754333"
Apr 16 16:30:10.412652 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:10.412612    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pdmjc" podStartSLOduration=2.113386244 podStartE2EDuration="19.412600661s" podCreationTimestamp="2026-04-16 16:29:51 +0000 UTC" firstStartedPulling="2026-04-16 16:29:52.493569674 +0000 UTC m=+1.890786042" lastFinishedPulling="2026-04-16 16:30:09.792784087 +0000 UTC m=+19.190000459" observedRunningTime="2026-04-16 16:30:10.412410793 +0000 UTC m=+19.809627182" watchObservedRunningTime="2026-04-16 16:30:10.412600661 +0000 UTC m=+19.809817049"
Apr 16 16:30:10.425158 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:10.425110    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.ec2.internal" podStartSLOduration=18.425093927 podStartE2EDuration="18.425093927s" podCreationTimestamp="2026-04-16 16:29:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:30:10.424175195 +0000 UTC m=+19.821391585" watchObservedRunningTime="2026-04-16 16:30:10.425093927 +0000 UTC m=+19.822310316"
Apr 16 16:30:11.394769 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:11.394739    2578 generic.go:358] "Generic (PLEG): container finished" podID="c3a26939a85c3397690ad7fddb03cc34" containerID="a4d7afac8fd03579104059d774d5741181de71e10ab1958a8d43cc4fd7fedf13" exitCode=0
Apr 16 16:30:11.395179 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:11.394818    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal" event={"ID":"c3a26939a85c3397690ad7fddb03cc34","Type":"ContainerDied","Data":"a4d7afac8fd03579104059d774d5741181de71e10ab1958a8d43cc4fd7fedf13"}
Apr 16 16:30:11.396242 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:11.396136    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rpnzb" event={"ID":"350fba86-71d9-4d96-b9ed-de22ab9333c9","Type":"ContainerStarted","Data":"deee2ae785a648f3151209d2383c50ad1817eb62590577aca80a94de75e547c0"}
Apr 16 16:30:11.397547 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:11.397523    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns" event={"ID":"158e69b7-65d9-4169-9fdb-cdb6e74eecb6","Type":"ContainerStarted","Data":"2fa3ee2dd9c3e1e0856867e9e860e99e6fdbdc1b6719c38d17da064cee3f2fa3"}
Apr 16 16:30:11.399978 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:11.399953    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7qft9" event={"ID":"d36c965f-65b5-4340-8151-e2ff609581b6","Type":"ContainerStarted","Data":"2c5c55da046db033be418b2bfdee16bd3b3af8c9def4ba66e86e125edd8bc0f1"}
Apr 16 16:30:11.402318 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:11.402296    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xdwz4" event={"ID":"d64db671-44fb-4c47-bda6-b8c720247f41","Type":"ContainerStarted","Data":"7ccd79e6747433fe8b7c68418d39ca64f1c5ee6f8f2dfa864ef05e5231c99d4e"}
Apr 16 16:30:11.403943 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:11.403923    2578 generic.go:358] "Generic (PLEG): container finished" podID="57674e7d-ee63-4a14-85d1-8c35f765106d" containerID="8e82e96c9de9a73fc7fcd973e227ab614b4f3f7d0afe8f10d6018fa4d4d353b5" exitCode=0
Apr 16 16:30:11.404041 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:11.403998    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnptn" event={"ID":"57674e7d-ee63-4a14-85d1-8c35f765106d","Type":"ContainerDied","Data":"8e82e96c9de9a73fc7fcd973e227ab614b4f3f7d0afe8f10d6018fa4d4d353b5"}
Apr 16 16:30:11.405426 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:11.405364    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fbgtp" event={"ID":"12c1b394-8723-4774-aecd-3c20c35fc722","Type":"ContainerStarted","Data":"1f8ba0e338b98061076e1237ed0c46a195e7cd159e04ef8cedacc3534d838464"}
Apr 16 16:30:11.425391 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:11.425356    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xdwz4" podStartSLOduration=3.138487983 podStartE2EDuration="20.425344817s" podCreationTimestamp="2026-04-16 16:29:51 +0000 UTC" firstStartedPulling="2026-04-16 16:29:52.479950589 +0000 UTC m=+1.877166955" lastFinishedPulling="2026-04-16 16:30:09.766807409 +0000 UTC m=+19.164023789" observedRunningTime="2026-04-16 16:30:11.424796938 +0000 UTC m=+20.822013352" watchObservedRunningTime="2026-04-16 16:30:11.425344817 +0000 UTC m=+20.822561205"
Apr 16 16:30:11.438092 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:11.437953    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7qft9" podStartSLOduration=3.169642451 podStartE2EDuration="20.437938035s" podCreationTimestamp="2026-04-16 16:29:51 +0000 UTC" firstStartedPulling="2026-04-16 16:29:52.498577435 +0000 UTC m=+1.895793804" lastFinishedPulling="2026-04-16 16:30:09.766873018 +0000 UTC m=+19.164089388" observedRunningTime="2026-04-16 16:30:11.437517874 +0000 UTC m=+20.834734261" watchObservedRunningTime="2026-04-16 16:30:11.437938035 +0000 UTC m=+20.835154423"
Apr 16 16:30:11.444738 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:11.444718    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Apr 16 16:30:11.450035 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:11.450000    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rpnzb" podStartSLOduration=3.19446402 podStartE2EDuration="20.44998585s" podCreationTimestamp="2026-04-16 16:29:51 +0000 UTC" firstStartedPulling="2026-04-16 16:29:52.511291892 +0000 UTC m=+1.908508258" lastFinishedPulling="2026-04-16 16:30:09.766813714 +0000 UTC m=+19.164030088" observedRunningTime="2026-04-16 16:30:11.449970586 +0000 UTC m=+20.847186974" watchObservedRunningTime="2026-04-16 16:30:11.44998585 +0000 UTC m=+20.847202238"
Apr 16 16:30:11.482720 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:11.482677    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-fbgtp" podStartSLOduration=3.132989029 podStartE2EDuration="20.482667324s" podCreationTimestamp="2026-04-16 16:29:51 +0000 UTC" firstStartedPulling="2026-04-16 16:29:52.427934379 +0000 UTC m=+1.825150746" lastFinishedPulling="2026-04-16 16:30:09.777612659 +0000 UTC m=+19.174829041" observedRunningTime="2026-04-16 16:30:11.462529741 +0000 UTC m=+20.859746129" watchObservedRunningTime="2026-04-16 16:30:11.482667324 +0000 UTC m=+20.879883712"
Apr 16 16:30:11.857179 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:11.857152    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rpnzb"
Apr 16 16:30:12.183275 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:12.183135    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-04-16T16:30:11.444735255Z","UUID":"3c456790-49c5-4833-88f9-7b7a1980469a","Handler":null,"Name":"","Endpoint":""}
Apr 16 16:30:12.184977 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:12.184954    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Apr 16 16:30:12.185121 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:12.184984    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Apr 16 16:30:12.231632 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:12.231606    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:12.231772 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:12.231606    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:12.231772 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:12.231716    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:30:12.231866 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:12.231800    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:30:12.409778 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:12.409743    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal" event={"ID":"c3a26939a85c3397690ad7fddb03cc34","Type":"ContainerStarted","Data":"96ab97efebbb0493ed8a60436f2f1c0f3ce592320a0282c1e33eaeed883150e5"}
Apr 16 16:30:12.412519 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:12.412494    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns" event={"ID":"158e69b7-65d9-4169-9fdb-cdb6e74eecb6","Type":"ContainerStarted","Data":"4fbcf1260012b0d5fa3e251a2770e3ce4524f4556a860b5e383fbec5297b3bf2"}
Apr 16 16:30:12.423169 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:12.423134    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.ec2.internal" podStartSLOduration=20.42312308 podStartE2EDuration="20.42312308s" podCreationTimestamp="2026-04-16 16:29:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:30:12.422836646 +0000 UTC m=+21.820053035" watchObservedRunningTime="2026-04-16 16:30:12.42312308 +0000 UTC m=+21.820339468"
Apr 16 16:30:12.447864 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:12.447704    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rpnzb"
Apr 16 16:30:12.448318 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:12.448298    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rpnzb"
Apr 16 16:30:13.415712 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:13.415677    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns" event={"ID":"158e69b7-65d9-4169-9fdb-cdb6e74eecb6","Type":"ContainerStarted","Data":"9fa8037cf196a156c7142e7a24ab81143912400fc9acb5b39d86a1061ee7f7d5"}
Apr 16 16:30:13.418704 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:13.418679    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 16:30:13.419134 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:13.419033    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld" event={"ID":"78b0d7e4-bab7-4f8d-ac1e-f45bede95414","Type":"ContainerStarted","Data":"fb8fc8dd92188a12bf19418da6b98d7bfbef7bed31c1343edbffe7fb6ed42d4b"}
Apr 16 16:30:13.419961 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:13.419942    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rpnzb"
Apr 16 16:30:13.433069 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:13.432984    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zl6ns" podStartSLOduration=2.134434907 podStartE2EDuration="22.432968232s" podCreationTimestamp="2026-04-16 16:29:51 +0000 UTC" firstStartedPulling="2026-04-16 16:29:52.485978873 +0000 UTC m=+1.883195244" lastFinishedPulling="2026-04-16 16:30:12.784512192 +0000 UTC m=+22.181728569" observedRunningTime="2026-04-16 16:30:13.432486404 +0000 UTC m=+22.829702816" watchObservedRunningTime="2026-04-16 16:30:13.432968232 +0000 UTC m=+22.830184622"
Apr 16 16:30:14.231423 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:14.231394    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:14.231608 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:14.231435    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:14.231608 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:14.231547    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:30:14.231716 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:14.231638    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:30:15.427350 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:15.427224    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 16:30:15.427905 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:15.427659    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld" event={"ID":"78b0d7e4-bab7-4f8d-ac1e-f45bede95414","Type":"ContainerStarted","Data":"0cf8ea6f16353cfebc5e704aff2febb366e29ccb963522b9257e41d88db87555"}
Apr 16 16:30:15.428133 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:15.428105    2578 scope.go:117] "RemoveContainer" containerID="0f15a80e9f8fd4977016c3686f6427d572d293c4ba61c269f5ca9e710d235dcf"
Apr 16 16:30:16.231019 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:16.230847    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:16.231152 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:16.230847    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:16.231152 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:16.231091    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:30:16.231221 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:16.231162    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:30:16.432128 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:16.432109    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 16:30:16.432545 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:16.432426    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld" event={"ID":"78b0d7e4-bab7-4f8d-ac1e-f45bede95414","Type":"ContainerStarted","Data":"c9cc7f18307df93b275d43459b0707e5e8d055f5217459bcb37ea32d6b9c29d6"}
Apr 16 16:30:16.432729 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:16.432710    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:30:16.432812 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:16.432738    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:30:16.432812 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:16.432752    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:30:16.434181 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:16.434156    2578 generic.go:358] "Generic (PLEG): container finished" podID="57674e7d-ee63-4a14-85d1-8c35f765106d" containerID="9243d90eb14d8e9fb19c63b78dde8c10eaec07db317b5e10f1b0a9c3571ebf81" exitCode=0
Apr 16 16:30:16.434279 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:16.434206    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnptn" event={"ID":"57674e7d-ee63-4a14-85d1-8c35f765106d","Type":"ContainerDied","Data":"9243d90eb14d8e9fb19c63b78dde8c10eaec07db317b5e10f1b0a9c3571ebf81"}
Apr 16 16:30:16.447271 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:16.447252    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:30:16.447361 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:16.447307    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:30:16.461929 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:16.459754    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld" podStartSLOduration=8.135342965 podStartE2EDuration="25.459464261s" podCreationTimestamp="2026-04-16 16:29:51 +0000 UTC" firstStartedPulling="2026-04-16 16:29:52.50663582 +0000 UTC m=+1.903852185" lastFinishedPulling="2026-04-16 16:30:09.830757113 +0000 UTC m=+19.227973481" observedRunningTime="2026-04-16 16:30:16.459043571 +0000 UTC m=+25.856259959" watchObservedRunningTime="2026-04-16 16:30:16.459464261 +0000 UTC m=+25.856680654"
Apr 16 16:30:18.230894 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:18.230864    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:18.231265 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:18.230874    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:18.231265 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:18.230960    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:30:18.231265 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:18.231059    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:30:18.439175 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:18.439152    2578 generic.go:358] "Generic (PLEG): container finished" podID="57674e7d-ee63-4a14-85d1-8c35f765106d" containerID="d7b1378026edb0bc1d82d2cd979cb4861e432c05d08b589d341a6c47eaf4c75c" exitCode=0
Apr 16 16:30:18.439312 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:18.439244    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnptn" event={"ID":"57674e7d-ee63-4a14-85d1-8c35f765106d","Type":"ContainerDied","Data":"d7b1378026edb0bc1d82d2cd979cb4861e432c05d08b589d341a6c47eaf4c75c"}
Apr 16 16:30:19.280147 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:19.280111    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wq65n"]
Apr 16 16:30:19.280529 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:19.280277    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:19.280529 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:19.280401    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:30:19.280922 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:19.280887    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w2mqw"]
Apr 16 16:30:19.281027 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:19.280989    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:19.281093 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:19.281071    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:30:19.443474 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:19.443446    2578 generic.go:358] "Generic (PLEG): container finished" podID="57674e7d-ee63-4a14-85d1-8c35f765106d" containerID="29042ce6da8007456703b2202bcdd942ba4b3f6a6f00bca052a955fb402071f4" exitCode=0
Apr 16 16:30:19.443620 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:19.443505    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnptn" event={"ID":"57674e7d-ee63-4a14-85d1-8c35f765106d","Type":"ContainerDied","Data":"29042ce6da8007456703b2202bcdd942ba4b3f6a6f00bca052a955fb402071f4"}
Apr 16 16:30:21.232152 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:21.232125    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:21.232524 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:21.232222    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:30:21.232524 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:21.232312    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:21.232524 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:21.232418    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:30:23.231094 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.231061    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:23.231531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.231068    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:23.231531 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:23.231194    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wq65n" podUID="21a9695d-812c-4ad6-9b18-ade579b343e0"
Apr 16 16:30:23.231531 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:23.231259    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w2mqw" podUID="29bd11bf-6683-4b59-9b71-e5f556a0ad04"
Apr 16 16:30:23.862953 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.862712    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs\") pod \"network-metrics-daemon-wq65n\" (UID: \"21a9695d-812c-4ad6-9b18-ade579b343e0\") " pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:23.863102 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:23.862867    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:30:23.863102 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:23.863019    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs podName:21a9695d-812c-4ad6-9b18-ade579b343e0 nodeName:}" failed. No retries permitted until 2026-04-16 16:30:55.863003139 +0000 UTC m=+65.260219528 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs") pod "network-metrics-daemon-wq65n" (UID: "21a9695d-812c-4ad6-9b18-ade579b343e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:30:23.915412 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.915383    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-164.ec2.internal" event="NodeReady"
Apr 16 16:30:23.915610 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.915502    2578 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
Apr 16 16:30:23.954982 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.954952    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jrfnm"]
Apr 16 16:30:23.958792 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.958755    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mvjtr"]
Apr 16 16:30:23.958943 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.958923    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jrfnm"
Apr 16 16:30:23.961141 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.961121    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8mbbv\""
Apr 16 16:30:23.961310 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.961128    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Apr 16 16:30:23.961506 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.961489    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Apr 16 16:30:23.961754 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.961736    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mvjtr"
Apr 16 16:30:23.963310 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.963288    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5wx\" (UniqueName: \"kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx\") pod \"network-check-target-w2mqw\" (UID: \"29bd11bf-6683-4b59-9b71-e5f556a0ad04\") " pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:23.963481 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:23.963431    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 16 16:30:23.963481 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:23.963454    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 16 16:30:23.963481 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:23.963468    2578 projected.go:194] Error preparing data for projected volume kube-api-access-zf5wx for pod openshift-network-diagnostics/network-check-target-w2mqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:30:23.963701 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:23.963537    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx podName:29bd11bf-6683-4b59-9b71-e5f556a0ad04 nodeName:}" failed. No retries permitted until 2026-04-16 16:30:55.963517259 +0000 UTC m=+65.360733628 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf5wx" (UniqueName: "kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx") pod "network-check-target-w2mqw" (UID: "29bd11bf-6683-4b59-9b71-e5f556a0ad04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:30:23.963902 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.963882    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Apr 16 16:30:23.964047 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.964016    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Apr 16 16:30:23.964146 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.964116    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mk2ls\""
Apr 16 16:30:23.964240 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.964214    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Apr 16 16:30:23.967087 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.967053    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jrfnm"]
Apr 16 16:30:23.969912 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:23.969891    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mvjtr"]
Apr 16 16:30:24.063777 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.063747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b569a454-1fb0-4aae-ace9-f8ea8f122839-cert\") pod \"ingress-canary-mvjtr\" (UID: \"b569a454-1fb0-4aae-ace9-f8ea8f122839\") " pod="openshift-ingress-canary/ingress-canary-mvjtr"
Apr 16 16:30:24.063930 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.063792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43bc80a-5182-4784-9565-41fe3bf16f60-tmp-dir\") pod \"dns-default-jrfnm\" (UID: \"f43bc80a-5182-4784-9565-41fe3bf16f60\") " pod="openshift-dns/dns-default-jrfnm"
Apr 16 16:30:24.063930 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.063857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f43bc80a-5182-4784-9565-41fe3bf16f60-config-volume\") pod \"dns-default-jrfnm\" (UID: \"f43bc80a-5182-4784-9565-41fe3bf16f60\") " pod="openshift-dns/dns-default-jrfnm"
Apr 16 16:30:24.063930 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.063879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f43bc80a-5182-4784-9565-41fe3bf16f60-metrics-tls\") pod \"dns-default-jrfnm\" (UID: \"f43bc80a-5182-4784-9565-41fe3bf16f60\") " pod="openshift-dns/dns-default-jrfnm"
Apr 16 16:30:24.063930 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.063894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtgss\" (UniqueName: \"kubernetes.io/projected/b569a454-1fb0-4aae-ace9-f8ea8f122839-kube-api-access-xtgss\") pod \"ingress-canary-mvjtr\" (UID: \"b569a454-1fb0-4aae-ace9-f8ea8f122839\") " pod="openshift-ingress-canary/ingress-canary-mvjtr"
Apr 16 16:30:24.063930 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.063911    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntgxx\" (UniqueName: \"kubernetes.io/projected/f43bc80a-5182-4784-9565-41fe3bf16f60-kube-api-access-ntgxx\") pod \"dns-default-jrfnm\" (UID: \"f43bc80a-5182-4784-9565-41fe3bf16f60\") " pod="openshift-dns/dns-default-jrfnm"
Apr 16 16:30:24.164998 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.164921    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f43bc80a-5182-4784-9565-41fe3bf16f60-config-volume\") pod \"dns-default-jrfnm\" (UID: \"f43bc80a-5182-4784-9565-41fe3bf16f60\") " pod="openshift-dns/dns-default-jrfnm"
Apr 16 16:30:24.164998 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.164966    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f43bc80a-5182-4784-9565-41fe3bf16f60-metrics-tls\") pod \"dns-default-jrfnm\" (UID: \"f43bc80a-5182-4784-9565-41fe3bf16f60\") " pod="openshift-dns/dns-default-jrfnm"
Apr 16 16:30:24.164998 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.164991    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xtgss\" (UniqueName: \"kubernetes.io/projected/b569a454-1fb0-4aae-ace9-f8ea8f122839-kube-api-access-xtgss\") pod \"ingress-canary-mvjtr\" (UID: \"b569a454-1fb0-4aae-ace9-f8ea8f122839\") " pod="openshift-ingress-canary/ingress-canary-mvjtr"
Apr 16 16:30:24.165204 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.165017    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ntgxx\" (UniqueName: \"kubernetes.io/projected/f43bc80a-5182-4784-9565-41fe3bf16f60-kube-api-access-ntgxx\") pod \"dns-default-jrfnm\" (UID: \"f43bc80a-5182-4784-9565-41fe3bf16f60\") " pod="openshift-dns/dns-default-jrfnm"
Apr 16 16:30:24.165534 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.165504    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b569a454-1fb0-4aae-ace9-f8ea8f122839-cert\") pod \"ingress-canary-mvjtr\" (UID: \"b569a454-1fb0-4aae-ace9-f8ea8f122839\") " pod="openshift-ingress-canary/ingress-canary-mvjtr"
Apr 16 16:30:24.165669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.165544    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43bc80a-5182-4784-9565-41fe3bf16f60-tmp-dir\") pod \"dns-default-jrfnm\" (UID: \"f43bc80a-5182-4784-9565-41fe3bf16f60\") " pod="openshift-dns/dns-default-jrfnm"
Apr 16 16:30:24.165669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.165579    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f43bc80a-5182-4784-9565-41fe3bf16f60-config-volume\") pod \"dns-default-jrfnm\" (UID: \"f43bc80a-5182-4784-9565-41fe3bf16f60\") " pod="openshift-dns/dns-default-jrfnm"
Apr 16 16:30:24.165807 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.165785    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43bc80a-5182-4784-9565-41fe3bf16f60-tmp-dir\") pod \"dns-default-jrfnm\" (UID: \"f43bc80a-5182-4784-9565-41fe3bf16f60\") " pod="openshift-dns/dns-default-jrfnm"
Apr 16 16:30:24.169433 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.169410    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f43bc80a-5182-4784-9565-41fe3bf16f60-metrics-tls\") pod \"dns-default-jrfnm\" (UID: \"f43bc80a-5182-4784-9565-41fe3bf16f60\") " pod="openshift-dns/dns-default-jrfnm"
Apr 16 16:30:24.169566 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.169449    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b569a454-1fb0-4aae-ace9-f8ea8f122839-cert\") pod \"ingress-canary-mvjtr\" (UID: \"b569a454-1fb0-4aae-ace9-f8ea8f122839\") " pod="openshift-ingress-canary/ingress-canary-mvjtr"
Apr 16 16:30:24.174415 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.174289    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtgss\" (UniqueName: \"kubernetes.io/projected/b569a454-1fb0-4aae-ace9-f8ea8f122839-kube-api-access-xtgss\") pod \"ingress-canary-mvjtr\" (UID: \"b569a454-1fb0-4aae-ace9-f8ea8f122839\") " pod="openshift-ingress-canary/ingress-canary-mvjtr"
Apr 16 16:30:24.174873 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.174849    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntgxx\" (UniqueName: \"kubernetes.io/projected/f43bc80a-5182-4784-9565-41fe3bf16f60-kube-api-access-ntgxx\") pod \"dns-default-jrfnm\" (UID: \"f43bc80a-5182-4784-9565-41fe3bf16f60\") " pod="openshift-dns/dns-default-jrfnm"
Apr 16 16:30:24.251267 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.251235    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-gh2cl"]
Apr 16 16:30:24.255535 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.255510    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-gh2cl"
Apr 16 16:30:24.257729 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.257707    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-v5dhs\""
Apr 16 16:30:24.257831 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.257737    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Apr 16 16:30:24.261183 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.261163    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-gh2cl"]
Apr 16 16:30:24.273079 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.273061    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jrfnm"
Apr 16 16:30:24.280094 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.280067    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mvjtr"
Apr 16 16:30:24.366691 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.366660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4388c084-9228-49d3-ac5a-e5e0951c6e83-tls-certificates\") pod \"prometheus-operator-admission-webhook-9cb97cd87-gh2cl\" (UID: \"4388c084-9228-49d3-ac5a-e5e0951c6e83\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-gh2cl"
Apr 16 16:30:24.467588 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.467549    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4388c084-9228-49d3-ac5a-e5e0951c6e83-tls-certificates\") pod \"prometheus-operator-admission-webhook-9cb97cd87-gh2cl\" (UID: \"4388c084-9228-49d3-ac5a-e5e0951c6e83\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-gh2cl"
Apr 16 16:30:24.470183 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.470155    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4388c084-9228-49d3-ac5a-e5e0951c6e83-tls-certificates\") pod \"prometheus-operator-admission-webhook-9cb97cd87-gh2cl\" (UID: \"4388c084-9228-49d3-ac5a-e5e0951c6e83\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-gh2cl"
Apr 16 16:30:24.566572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.566526    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-gh2cl"
Apr 16 16:30:24.939914 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.939822    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-586b57c7b4-fxvzp"]
Apr 16 16:30:24.942940 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.942917    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-586b57c7b4-fxvzp"
Apr 16 16:30:24.945606 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.945584    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zm29w\""
Apr 16 16:30:24.945709 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.945584    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Apr 16 16:30:24.945709 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.945584    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Apr 16 16:30:24.954030 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:24.954007    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-586b57c7b4-fxvzp"]
Apr 16 16:30:25.071498 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:25.071465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvwsm\" (UniqueName: \"kubernetes.io/projected/b99ebca1-5a6b-44b9-8db2-e29fad719f8b-kube-api-access-kvwsm\") pod \"downloads-586b57c7b4-fxvzp\" (UID: \"b99ebca1-5a6b-44b9-8db2-e29fad719f8b\") " pod="openshift-console/downloads-586b57c7b4-fxvzp"
Apr 16 16:30:25.172009 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:25.171971    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kvwsm\" (UniqueName: \"kubernetes.io/projected/b99ebca1-5a6b-44b9-8db2-e29fad719f8b-kube-api-access-kvwsm\") pod \"downloads-586b57c7b4-fxvzp\" (UID: \"b99ebca1-5a6b-44b9-8db2-e29fad719f8b\") " pod="openshift-console/downloads-586b57c7b4-fxvzp"
Apr 16 16:30:25.179398 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:25.179372    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvwsm\" (UniqueName: \"kubernetes.io/projected/b99ebca1-5a6b-44b9-8db2-e29fad719f8b-kube-api-access-kvwsm\") pod \"downloads-586b57c7b4-fxvzp\" (UID: \"b99ebca1-5a6b-44b9-8db2-e29fad719f8b\") " pod="openshift-console/downloads-586b57c7b4-fxvzp"
Apr 16 16:30:25.231371 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:25.231349    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:25.231613 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:25.231593    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:25.233995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:25.233969    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tqp8k\""
Apr 16 16:30:25.234107 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:25.234009    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Apr 16 16:30:25.234107 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:25.234028    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Apr 16 16:30:25.234107 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:25.234084    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ws7rw\""
Apr 16 16:30:25.234318 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:25.234300    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Apr 16 16:30:25.253149 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:25.253133    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-586b57c7b4-fxvzp"
Apr 16 16:30:25.834481 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:25.834257    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mvjtr"]
Apr 16 16:30:25.836216 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:25.836159    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-586b57c7b4-fxvzp"]
Apr 16 16:30:25.837068 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:25.837040    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jrfnm"]
Apr 16 16:30:25.837906 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:25.837754    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-gh2cl"]
Apr 16 16:30:25.949756 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:30:25.949714    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99ebca1_5a6b_44b9_8db2_e29fad719f8b.slice/crio-dfd98b057e99cd615b43dec42e6efb723d4a761183f8684669867c4df3556fcf WatchSource:0}: Error finding container dfd98b057e99cd615b43dec42e6efb723d4a761183f8684669867c4df3556fcf: Status 404 returned error can't find the container with id dfd98b057e99cd615b43dec42e6efb723d4a761183f8684669867c4df3556fcf
Apr 16 16:30:25.951514 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:30:25.950586    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43bc80a_5182_4784_9565_41fe3bf16f60.slice/crio-cd78f8cae6e2f81eea8a00c4c6490e30bb94ad7c851ae329d50e6889d583abba WatchSource:0}: Error finding container cd78f8cae6e2f81eea8a00c4c6490e30bb94ad7c851ae329d50e6889d583abba: Status 404 returned error can't find the container with id cd78f8cae6e2f81eea8a00c4c6490e30bb94ad7c851ae329d50e6889d583abba
Apr 16 16:30:25.951693 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:30:25.951666    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4388c084_9228_49d3_ac5a_e5e0951c6e83.slice/crio-2fb74566ea364a77f6920a48c2494f4e694d02a720aedfd16c702dd2abbb6817 WatchSource:0}: Error finding container 2fb74566ea364a77f6920a48c2494f4e694d02a720aedfd16c702dd2abbb6817: Status 404 returned error can't find the container with id 2fb74566ea364a77f6920a48c2494f4e694d02a720aedfd16c702dd2abbb6817
Apr 16 16:30:25.952072 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:30:25.952044    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb569a454_1fb0_4aae_ace9_f8ea8f122839.slice/crio-25bf3f3348722e94fee791e6e15b6bb5c86a27cddba48fbcbc27b1306b8c47d4 WatchSource:0}: Error finding container 25bf3f3348722e94fee791e6e15b6bb5c86a27cddba48fbcbc27b1306b8c47d4: Status 404 returned error can't find the container with id 25bf3f3348722e94fee791e6e15b6bb5c86a27cddba48fbcbc27b1306b8c47d4
Apr 16 16:30:26.457536 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:26.457503    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mvjtr" event={"ID":"b569a454-1fb0-4aae-ace9-f8ea8f122839","Type":"ContainerStarted","Data":"25bf3f3348722e94fee791e6e15b6bb5c86a27cddba48fbcbc27b1306b8c47d4"}
Apr 16 16:30:26.459788 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:26.459763    2578 generic.go:358] "Generic (PLEG): container finished" podID="57674e7d-ee63-4a14-85d1-8c35f765106d" containerID="ade1e7edcf9bc11b84537f9f56173551ca44dfd9fad05f7f91b65e8c3ed2a59d" exitCode=0
Apr 16 16:30:26.459917 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:26.459798    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnptn" event={"ID":"57674e7d-ee63-4a14-85d1-8c35f765106d","Type":"ContainerDied","Data":"ade1e7edcf9bc11b84537f9f56173551ca44dfd9fad05f7f91b65e8c3ed2a59d"}
Apr 16 16:30:26.460845 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:26.460815    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jrfnm" event={"ID":"f43bc80a-5182-4784-9565-41fe3bf16f60","Type":"ContainerStarted","Data":"cd78f8cae6e2f81eea8a00c4c6490e30bb94ad7c851ae329d50e6889d583abba"}
Apr 16 16:30:26.461858 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:26.461838    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-586b57c7b4-fxvzp" event={"ID":"b99ebca1-5a6b-44b9-8db2-e29fad719f8b","Type":"ContainerStarted","Data":"dfd98b057e99cd615b43dec42e6efb723d4a761183f8684669867c4df3556fcf"}
Apr 16 16:30:26.462849 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:26.462825    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-gh2cl" event={"ID":"4388c084-9228-49d3-ac5a-e5e0951c6e83","Type":"ContainerStarted","Data":"2fb74566ea364a77f6920a48c2494f4e694d02a720aedfd16c702dd2abbb6817"}
Apr 16 16:30:27.172772 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:27.172652    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fbgtp_12c1b394-8723-4774-aecd-3c20c35fc722/dns-node-resolver/0.log"
Apr 16 16:30:27.469651 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:27.469612    2578 generic.go:358] "Generic (PLEG): container finished" podID="57674e7d-ee63-4a14-85d1-8c35f765106d" containerID="585cbc28a31ec1b3a6aadc1d8ffa046db9cf4d50cee700ec19b918d0f43b1c86" exitCode=0
Apr 16 16:30:27.470083 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:27.469677    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnptn" event={"ID":"57674e7d-ee63-4a14-85d1-8c35f765106d","Type":"ContainerDied","Data":"585cbc28a31ec1b3a6aadc1d8ffa046db9cf4d50cee700ec19b918d0f43b1c86"}
Apr 16 16:30:27.963741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:27.963709    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j72gz"]
Apr 16 16:30:27.976648 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:27.976623    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j72gz"]
Apr 16 16:30:27.976784 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:27.976756    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:27.979446 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:27.979424    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7qft9_d36c965f-65b5-4340-8151-e2ff609581b6/node-ca/0.log"
Apr 16 16:30:27.980754 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:27.979888    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tj77f\""
Apr 16 16:30:27.980754 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:27.980163    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Apr 16 16:30:27.980754 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:27.980386    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Apr 16 16:30:27.980754 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:27.980592    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Apr 16 16:30:27.980754 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:27.980612    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Apr 16 16:30:28.094911 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.094875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb5mj\" (UniqueName: \"kubernetes.io/projected/9d15723d-383a-4598-907c-bb0f4224279d-kube-api-access-wb5mj\") pod \"insights-runtime-extractor-j72gz\" (UID: \"9d15723d-383a-4598-907c-bb0f4224279d\") " pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:28.095058 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.094949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d15723d-383a-4598-907c-bb0f4224279d-data-volume\") pod \"insights-runtime-extractor-j72gz\" (UID: \"9d15723d-383a-4598-907c-bb0f4224279d\") " pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:28.095058 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.095013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d15723d-383a-4598-907c-bb0f4224279d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j72gz\" (UID: \"9d15723d-383a-4598-907c-bb0f4224279d\") " pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:28.095172 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.095056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d15723d-383a-4598-907c-bb0f4224279d-crio-socket\") pod \"insights-runtime-extractor-j72gz\" (UID: \"9d15723d-383a-4598-907c-bb0f4224279d\") " pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:28.095172 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.095082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d15723d-383a-4598-907c-bb0f4224279d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j72gz\" (UID: \"9d15723d-383a-4598-907c-bb0f4224279d\") " pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:28.195484 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.195450    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d15723d-383a-4598-907c-bb0f4224279d-data-volume\") pod \"insights-runtime-extractor-j72gz\" (UID: \"9d15723d-383a-4598-907c-bb0f4224279d\") " pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:28.195654 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.195499    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d15723d-383a-4598-907c-bb0f4224279d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j72gz\" (UID: \"9d15723d-383a-4598-907c-bb0f4224279d\") " pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:28.195654 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.195533    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d15723d-383a-4598-907c-bb0f4224279d-crio-socket\") pod \"insights-runtime-extractor-j72gz\" (UID: \"9d15723d-383a-4598-907c-bb0f4224279d\") " pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:28.195654 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.195580    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d15723d-383a-4598-907c-bb0f4224279d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j72gz\" (UID: \"9d15723d-383a-4598-907c-bb0f4224279d\") " pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:28.195802 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.195735    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wb5mj\" (UniqueName: \"kubernetes.io/projected/9d15723d-383a-4598-907c-bb0f4224279d-kube-api-access-wb5mj\") pod \"insights-runtime-extractor-j72gz\" (UID: \"9d15723d-383a-4598-907c-bb0f4224279d\") " pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:28.195853 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.195811    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d15723d-383a-4598-907c-bb0f4224279d-crio-socket\") pod \"insights-runtime-extractor-j72gz\" (UID: \"9d15723d-383a-4598-907c-bb0f4224279d\") " pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:28.195853 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.195823    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d15723d-383a-4598-907c-bb0f4224279d-data-volume\") pod \"insights-runtime-extractor-j72gz\" (UID: \"9d15723d-383a-4598-907c-bb0f4224279d\") " pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:28.201984 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.201960    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d15723d-383a-4598-907c-bb0f4224279d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j72gz\" (UID: \"9d15723d-383a-4598-907c-bb0f4224279d\") " pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:28.205398 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.205373    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d15723d-383a-4598-907c-bb0f4224279d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j72gz\" (UID: \"9d15723d-383a-4598-907c-bb0f4224279d\") " pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:28.205398 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.205387    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb5mj\" (UniqueName: \"kubernetes.io/projected/9d15723d-383a-4598-907c-bb0f4224279d-kube-api-access-wb5mj\") pod \"insights-runtime-extractor-j72gz\" (UID: \"9d15723d-383a-4598-907c-bb0f4224279d\") " pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:28.289887 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.289812    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j72gz"
Apr 16 16:30:28.480012 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.479771    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnptn" event={"ID":"57674e7d-ee63-4a14-85d1-8c35f765106d","Type":"ContainerStarted","Data":"05275b47e43ce64c1fc2658af2efe034e3d958c41cd012483ac19746fda5bdf7"}
Apr 16 16:30:28.502193 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:28.502143    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jnptn" podStartSLOduration=3.959180318 podStartE2EDuration="37.502126717s" podCreationTimestamp="2026-04-16 16:29:51 +0000 UTC" firstStartedPulling="2026-04-16 16:29:52.459187978 +0000 UTC m=+1.856404344" lastFinishedPulling="2026-04-16 16:30:26.002134378 +0000 UTC m=+35.399350743" observedRunningTime="2026-04-16 16:30:28.501046315 +0000 UTC m=+37.898262704" watchObservedRunningTime="2026-04-16 16:30:28.502126717 +0000 UTC m=+37.899343107"
Apr 16 16:30:29.246778 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:29.246718    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j72gz"]
Apr 16 16:30:29.250404 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:30:29.250332    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d15723d_383a_4598_907c_bb0f4224279d.slice/crio-6232be0a5cc726c01be01a75911e1826917c6a2c757aee63295ce1dba0039984 WatchSource:0}: Error finding container 6232be0a5cc726c01be01a75911e1826917c6a2c757aee63295ce1dba0039984: Status 404 returned error can't find the container with id 6232be0a5cc726c01be01a75911e1826917c6a2c757aee63295ce1dba0039984
Apr 16 16:30:29.485335 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:29.485265    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j72gz" event={"ID":"9d15723d-383a-4598-907c-bb0f4224279d","Type":"ContainerStarted","Data":"d091e87534fe203303ad2be8efc617e06eaae497bf0eeb6169d3aeebb6bdeaa4"}
Apr 16 16:30:29.485335 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:29.485313    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j72gz" event={"ID":"9d15723d-383a-4598-907c-bb0f4224279d","Type":"ContainerStarted","Data":"6232be0a5cc726c01be01a75911e1826917c6a2c757aee63295ce1dba0039984"}
Apr 16 16:30:29.487033 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:29.486975    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jrfnm" event={"ID":"f43bc80a-5182-4784-9565-41fe3bf16f60","Type":"ContainerStarted","Data":"67e4138c4d8435982e77b52f5c18906391e0659fd15c3a82ae5fc9cede7123fb"}
Apr 16 16:30:29.488371 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:29.488318    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-gh2cl" event={"ID":"4388c084-9228-49d3-ac5a-e5e0951c6e83","Type":"ContainerStarted","Data":"84cf049055f85660fb1acc4e4b6ce10edcdf1ae34481fd91c73180557ee1d0ca"}
Apr 16 16:30:29.488804 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:29.488717    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-gh2cl"
Apr 16 16:30:29.490145 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:29.490098    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mvjtr" event={"ID":"b569a454-1fb0-4aae-ace9-f8ea8f122839","Type":"ContainerStarted","Data":"a8010618c3a40e5e5e9ebdd1599044fea29d8edfff1355ee2614f2b7855e5545"}
Apr 16 16:30:29.495201 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:29.495178    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-gh2cl"
Apr 16 16:30:29.503774 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:29.503739    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-gh2cl" podStartSLOduration=2.365992226 podStartE2EDuration="5.503729893s" podCreationTimestamp="2026-04-16 16:30:24 +0000 UTC" firstStartedPulling="2026-04-16 16:30:25.978971773 +0000 UTC m=+35.376188139" lastFinishedPulling="2026-04-16 16:30:29.116709435 +0000 UTC m=+38.513925806" observedRunningTime="2026-04-16 16:30:29.503061964 +0000 UTC m=+38.900278353" watchObservedRunningTime="2026-04-16 16:30:29.503729893 +0000 UTC m=+38.900946281"
Apr 16 16:30:29.529628 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:29.529579    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mvjtr" podStartSLOduration=3.391795995 podStartE2EDuration="6.529547692s" podCreationTimestamp="2026-04-16 16:30:23 +0000 UTC" firstStartedPulling="2026-04-16 16:30:25.979111414 +0000 UTC m=+35.376327780" lastFinishedPulling="2026-04-16 16:30:29.116863097 +0000 UTC m=+38.514079477" observedRunningTime="2026-04-16 16:30:29.528815364 +0000 UTC m=+38.926031753" watchObservedRunningTime="2026-04-16 16:30:29.529547692 +0000 UTC m=+38.926764060"
Apr 16 16:30:30.315120 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.315090    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-78f957474d-lbm2d"]
Apr 16 16:30:30.318083 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.318057    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d"
Apr 16 16:30:30.320274 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.320251    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Apr 16 16:30:30.320274 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.320269    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bmc6w\""
Apr 16 16:30:30.320449 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.320320    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Apr 16 16:30:30.321184 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.321168    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Apr 16 16:30:30.321351 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.321329    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Apr 16 16:30:30.321438 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.321379    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Apr 16 16:30:30.325087 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.325065    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-78f957474d-lbm2d"]
Apr 16 16:30:30.414658 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.414598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c617adf1-5e8f-4c9c-8755-540fc1fcb407-prometheus-operator-tls\") pod \"prometheus-operator-78f957474d-lbm2d\" (UID: \"c617adf1-5e8f-4c9c-8755-540fc1fcb407\") " pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d"
Apr 16 16:30:30.414658 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.414630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c617adf1-5e8f-4c9c-8755-540fc1fcb407-metrics-client-ca\") pod \"prometheus-operator-78f957474d-lbm2d\" (UID: \"c617adf1-5e8f-4c9c-8755-540fc1fcb407\") " pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d"
Apr 16 16:30:30.414658 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.414647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbj6l\" (UniqueName: \"kubernetes.io/projected/c617adf1-5e8f-4c9c-8755-540fc1fcb407-kube-api-access-nbj6l\") pod \"prometheus-operator-78f957474d-lbm2d\" (UID: \"c617adf1-5e8f-4c9c-8755-540fc1fcb407\") " pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d"
Apr 16 16:30:30.414879 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.414744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c617adf1-5e8f-4c9c-8755-540fc1fcb407-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-78f957474d-lbm2d\" (UID: \"c617adf1-5e8f-4c9c-8755-540fc1fcb407\") " pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d"
Apr 16 16:30:30.495411 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.495377    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j72gz" event={"ID":"9d15723d-383a-4598-907c-bb0f4224279d","Type":"ContainerStarted","Data":"41cad25e99917737b17b567120c0ae17a1a2f667cc61d9ba7ff052f8c355a2d0"}
Apr 16 16:30:30.497176 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.497147    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jrfnm" event={"ID":"f43bc80a-5182-4784-9565-41fe3bf16f60","Type":"ContainerStarted","Data":"5717b6dd1b63e39182f9dd7c03948633e3c5501fb3d15c4a0eb52d3e9669b0d3"}
Apr 16 16:30:30.511903 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.511861    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-jrfnm" podStartSLOduration=4.371330419 podStartE2EDuration="7.511848666s" podCreationTimestamp="2026-04-16 16:30:23 +0000 UTC" firstStartedPulling="2026-04-16 16:30:25.978979515 +0000 UTC m=+35.376195881" lastFinishedPulling="2026-04-16 16:30:29.119497748 +0000 UTC m=+38.516714128" observedRunningTime="2026-04-16 16:30:30.510881072 +0000 UTC m=+39.908097463" watchObservedRunningTime="2026-04-16 16:30:30.511848666 +0000 UTC m=+39.909065054"
Apr 16 16:30:30.515132 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.515105    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c617adf1-5e8f-4c9c-8755-540fc1fcb407-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-78f957474d-lbm2d\" (UID: \"c617adf1-5e8f-4c9c-8755-540fc1fcb407\") " pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d"
Apr 16 16:30:30.515243 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.515182    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c617adf1-5e8f-4c9c-8755-540fc1fcb407-prometheus-operator-tls\") pod \"prometheus-operator-78f957474d-lbm2d\" (UID: \"c617adf1-5e8f-4c9c-8755-540fc1fcb407\") " pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d"
Apr 16 16:30:30.515243 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.515216    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c617adf1-5e8f-4c9c-8755-540fc1fcb407-metrics-client-ca\") pod \"prometheus-operator-78f957474d-lbm2d\" (UID: \"c617adf1-5e8f-4c9c-8755-540fc1fcb407\") " pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d"
Apr 16 16:30:30.515357 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.515242    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nbj6l\" (UniqueName: \"kubernetes.io/projected/c617adf1-5e8f-4c9c-8755-540fc1fcb407-kube-api-access-nbj6l\") pod \"prometheus-operator-78f957474d-lbm2d\" (UID: \"c617adf1-5e8f-4c9c-8755-540fc1fcb407\") " pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d"
Apr 16 16:30:30.515357 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:30.515335    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Apr 16 16:30:30.515450 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:30:30.515396    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c617adf1-5e8f-4c9c-8755-540fc1fcb407-prometheus-operator-tls podName:c617adf1-5e8f-4c9c-8755-540fc1fcb407 nodeName:}" failed. No retries permitted until 2026-04-16 16:30:31.015377855 +0000 UTC m=+40.412594227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c617adf1-5e8f-4c9c-8755-540fc1fcb407-prometheus-operator-tls") pod "prometheus-operator-78f957474d-lbm2d" (UID: "c617adf1-5e8f-4c9c-8755-540fc1fcb407") : secret "prometheus-operator-tls" not found
Apr 16 16:30:30.515983 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.515960    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c617adf1-5e8f-4c9c-8755-540fc1fcb407-metrics-client-ca\") pod \"prometheus-operator-78f957474d-lbm2d\" (UID: \"c617adf1-5e8f-4c9c-8755-540fc1fcb407\") " pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d"
Apr 16 16:30:30.518467 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.518440    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c617adf1-5e8f-4c9c-8755-540fc1fcb407-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-78f957474d-lbm2d\" (UID: \"c617adf1-5e8f-4c9c-8755-540fc1fcb407\") " pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d"
Apr 16 16:30:30.523657 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:30.523615    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbj6l\" (UniqueName: \"kubernetes.io/projected/c617adf1-5e8f-4c9c-8755-540fc1fcb407-kube-api-access-nbj6l\") pod \"prometheus-operator-78f957474d-lbm2d\" (UID: \"c617adf1-5e8f-4c9c-8755-540fc1fcb407\") " pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d"
Apr 16 16:30:31.021183 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:31.021150    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c617adf1-5e8f-4c9c-8755-540fc1fcb407-prometheus-operator-tls\") pod \"prometheus-operator-78f957474d-lbm2d\" (UID: \"c617adf1-5e8f-4c9c-8755-540fc1fcb407\") " pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d"
Apr 16 16:30:31.024248 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:31.024190    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c617adf1-5e8f-4c9c-8755-540fc1fcb407-prometheus-operator-tls\") pod \"prometheus-operator-78f957474d-lbm2d\" (UID: \"c617adf1-5e8f-4c9c-8755-540fc1fcb407\") " pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d"
Apr 16 16:30:31.228463 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:31.228433    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d"
Apr 16 16:30:31.500993 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:31.500950    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jrfnm"
Apr 16 16:30:31.835291 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:31.835263    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-78f957474d-lbm2d"]
Apr 16 16:30:31.840770 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:30:31.840745    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc617adf1_5e8f_4c9c_8755_540fc1fcb407.slice/crio-510b3597eaf21f4caad3a720aabc7c1b123d140e2de5352b24a35f2223dc3699 WatchSource:0}: Error finding container 510b3597eaf21f4caad3a720aabc7c1b123d140e2de5352b24a35f2223dc3699: Status 404 returned error can't find the container with id 510b3597eaf21f4caad3a720aabc7c1b123d140e2de5352b24a35f2223dc3699
Apr 16 16:30:32.505158 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:32.505118    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j72gz" event={"ID":"9d15723d-383a-4598-907c-bb0f4224279d","Type":"ContainerStarted","Data":"515663137ab43a228dd0dfcdc4017371f9dfe04db9fad884ddc831e1f7d0e123"}
Apr 16 16:30:32.506339 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:32.506304    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d" event={"ID":"c617adf1-5e8f-4c9c-8755-540fc1fcb407","Type":"ContainerStarted","Data":"510b3597eaf21f4caad3a720aabc7c1b123d140e2de5352b24a35f2223dc3699"}
Apr 16 16:30:32.522532 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:32.522480    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-j72gz" podStartSLOduration=3.122640757 podStartE2EDuration="5.522464465s" podCreationTimestamp="2026-04-16 16:30:27 +0000 UTC" firstStartedPulling="2026-04-16 16:30:29.371589707 +0000 UTC m=+38.768806075" lastFinishedPulling="2026-04-16 16:30:31.771413402 +0000 UTC m=+41.168629783" observedRunningTime="2026-04-16 16:30:32.521733632 +0000 UTC m=+41.918950045" watchObservedRunningTime="2026-04-16 16:30:32.522464465 +0000 UTC m=+41.919680855"
Apr 16 16:30:33.510598 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:33.510543    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d" event={"ID":"c617adf1-5e8f-4c9c-8755-540fc1fcb407","Type":"ContainerStarted","Data":"f1aa53cb83ed31ab0779508523e49df7c1413783b55b97eedd32c9f9f0c21945"}
Apr 16 16:30:33.510989 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:33.510603    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d" event={"ID":"c617adf1-5e8f-4c9c-8755-540fc1fcb407","Type":"ContainerStarted","Data":"372d93ade374ffb7ad9c7aff1b8d52b9d4e7ce5bcb058c549ba9699db24a4aeb"}
Apr 16 16:30:33.526659 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:33.526617    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-78f957474d-lbm2d" podStartSLOduration=2.234619599 podStartE2EDuration="3.526604207s" podCreationTimestamp="2026-04-16 16:30:30 +0000 UTC" firstStartedPulling="2026-04-16 16:30:31.843038424 +0000 UTC m=+41.240254793" lastFinishedPulling="2026-04-16 16:30:33.13502281 +0000 UTC m=+42.532239401" observedRunningTime="2026-04-16 16:30:33.525084551 +0000 UTC m=+42.922300939" watchObservedRunningTime="2026-04-16 16:30:33.526604207 +0000 UTC m=+42.923820593"
Apr 16 16:30:34.263396 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.261697    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c4b67854d-nv7hr"]
Apr 16 16:30:34.265117 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.265091    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.268513 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.268482    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Apr 16 16:30:34.268628 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.268535    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Apr 16 16:30:34.268628 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.268593    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z98rd\""
Apr 16 16:30:34.268628 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.268601    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Apr 16 16:30:34.268816 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.268537    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Apr 16 16:30:34.268816 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.268483    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Apr 16 16:30:34.273070 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.272922    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c4b67854d-nv7hr"]
Apr 16 16:30:34.446864 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.446836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-console-config\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.446864 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.446867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlrx4\" (UniqueName: \"kubernetes.io/projected/cde4edfa-4c1f-4006-8f70-12a437387527-kube-api-access-hlrx4\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.447090 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.446900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-service-ca\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.447090 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.446977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-oauth-serving-cert\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.447090 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.447009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cde4edfa-4c1f-4006-8f70-12a437387527-console-serving-cert\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.447090 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.447026    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cde4edfa-4c1f-4006-8f70-12a437387527-console-oauth-config\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.547759 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.547690    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cde4edfa-4c1f-4006-8f70-12a437387527-console-serving-cert\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.547759 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.547720    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cde4edfa-4c1f-4006-8f70-12a437387527-console-oauth-config\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.548237 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.547761    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-console-config\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.548237 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.547789    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hlrx4\" (UniqueName: \"kubernetes.io/projected/cde4edfa-4c1f-4006-8f70-12a437387527-kube-api-access-hlrx4\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.548237 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.547822    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-service-ca\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.548237 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.547881    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-oauth-serving-cert\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.548548 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.548503    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-console-config\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.548741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.548720    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-service-ca\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.549367 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.549342    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-oauth-serving-cert\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.550427 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.550402    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cde4edfa-4c1f-4006-8f70-12a437387527-console-oauth-config\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.550636 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.550610    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cde4edfa-4c1f-4006-8f70-12a437387527-console-serving-cert\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.555872 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.555849    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlrx4\" (UniqueName: \"kubernetes.io/projected/cde4edfa-4c1f-4006-8f70-12a437387527-kube-api-access-hlrx4\") pod \"console-7c4b67854d-nv7hr\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") " pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.575980 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.575960    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:34.705122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:34.705093    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c4b67854d-nv7hr"]
Apr 16 16:30:34.708830 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:30:34.708804    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde4edfa_4c1f_4006_8f70_12a437387527.slice/crio-52705f0816b95fd60a6355f79a4353d8d5178dc0665991043a384857c412dbb3 WatchSource:0}: Error finding container 52705f0816b95fd60a6355f79a4353d8d5178dc0665991043a384857c412dbb3: Status 404 returned error can't find the container with id 52705f0816b95fd60a6355f79a4353d8d5178dc0665991043a384857c412dbb3
Apr 16 16:30:35.516564 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.516504    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4b67854d-nv7hr" event={"ID":"cde4edfa-4c1f-4006-8f70-12a437387527","Type":"ContainerStarted","Data":"52705f0816b95fd60a6355f79a4353d8d5178dc0665991043a384857c412dbb3"}
Apr 16 16:30:35.660246 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.660218    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"]
Apr 16 16:30:35.684836 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.684811    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"]
Apr 16 16:30:35.684998 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.684960    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"
Apr 16 16:30:35.687650 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.687619    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Apr 16 16:30:35.687758 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.687650    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gsp26\""
Apr 16 16:30:35.687809 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.687767    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Apr 16 16:30:35.700584 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.700549    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s6mx4"]
Apr 16 16:30:35.724189 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.724170    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.726801 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.726783    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Apr 16 16:30:35.726899 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.726851    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Apr 16 16:30:35.727763 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.727545    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Apr 16 16:30:35.728743 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.728724    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-95zfb\""
Apr 16 16:30:35.857797 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.857723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f076b615-76d7-4867-ae84-0374653e85eb-root\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.857797 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.857768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f076b615-76d7-4867-ae84-0374653e85eb-node-exporter-tls\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.857995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.857808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a-metrics-client-ca\") pod \"openshift-state-metrics-5669946b84-cjq55\" (UID: \"ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"
Apr 16 16:30:35.857995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.857863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f076b615-76d7-4867-ae84-0374653e85eb-sys\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.857995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.857907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-5669946b84-cjq55\" (UID: \"ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"
Apr 16 16:30:35.857995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.857949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f076b615-76d7-4867-ae84-0374653e85eb-metrics-client-ca\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.857995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.857981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f076b615-76d7-4867-ae84-0374653e85eb-node-exporter-textfile\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.858195 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.858007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f076b615-76d7-4867-ae84-0374653e85eb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.858195 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.858058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f076b615-76d7-4867-ae84-0374653e85eb-node-exporter-wtmp\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.858195 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.858134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vspxs\" (UniqueName: \"kubernetes.io/projected/f076b615-76d7-4867-ae84-0374653e85eb-kube-api-access-vspxs\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.858195 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.858170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctlrq\" (UniqueName: \"kubernetes.io/projected/ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a-kube-api-access-ctlrq\") pod \"openshift-state-metrics-5669946b84-cjq55\" (UID: \"ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"
Apr 16 16:30:35.858348 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.858205    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-5669946b84-cjq55\" (UID: \"ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"
Apr 16 16:30:35.858348 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.858241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f076b615-76d7-4867-ae84-0374653e85eb-node-exporter-accelerators-collector-config\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.959451 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.959420    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vspxs\" (UniqueName: \"kubernetes.io/projected/f076b615-76d7-4867-ae84-0374653e85eb-kube-api-access-vspxs\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.959680 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.959468    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ctlrq\" (UniqueName: \"kubernetes.io/projected/ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a-kube-api-access-ctlrq\") pod \"openshift-state-metrics-5669946b84-cjq55\" (UID: \"ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"
Apr 16 16:30:35.959680 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.959531    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-5669946b84-cjq55\" (UID: \"ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"
Apr 16 16:30:35.959680 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.959590    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f076b615-76d7-4867-ae84-0374653e85eb-node-exporter-accelerators-collector-config\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.959680 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.959670    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f076b615-76d7-4867-ae84-0374653e85eb-root\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.959896 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.959704    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f076b615-76d7-4867-ae84-0374653e85eb-node-exporter-tls\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.959896 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.959740    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a-metrics-client-ca\") pod \"openshift-state-metrics-5669946b84-cjq55\" (UID: \"ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"
Apr 16 16:30:35.959896 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.959774    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f076b615-76d7-4867-ae84-0374653e85eb-sys\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.959896 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.959775    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f076b615-76d7-4867-ae84-0374653e85eb-root\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.959896 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.959810    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-5669946b84-cjq55\" (UID: \"ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"
Apr 16 16:30:35.959896 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.959850    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f076b615-76d7-4867-ae84-0374653e85eb-metrics-client-ca\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.959896 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.959879    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f076b615-76d7-4867-ae84-0374653e85eb-node-exporter-textfile\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.960115 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.959903    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f076b615-76d7-4867-ae84-0374653e85eb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.960115 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.959929    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f076b615-76d7-4867-ae84-0374653e85eb-node-exporter-wtmp\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.960115 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.960081    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f076b615-76d7-4867-ae84-0374653e85eb-node-exporter-wtmp\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.960472 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.960446    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f076b615-76d7-4867-ae84-0374653e85eb-sys\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.960599 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.960530    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f076b615-76d7-4867-ae84-0374653e85eb-metrics-client-ca\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.960599 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.960536    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a-metrics-client-ca\") pod \"openshift-state-metrics-5669946b84-cjq55\" (UID: \"ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"
Apr 16 16:30:35.961023 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.960965    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f076b615-76d7-4867-ae84-0374653e85eb-node-exporter-accelerators-collector-config\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.962597 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.962572    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f076b615-76d7-4867-ae84-0374653e85eb-node-exporter-tls\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.962735 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.962710    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-5669946b84-cjq55\" (UID: \"ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"
Apr 16 16:30:35.963271 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.963229    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f076b615-76d7-4867-ae84-0374653e85eb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.963827 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.963806    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-5669946b84-cjq55\" (UID: \"ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"
Apr 16 16:30:35.968154 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.968130    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vspxs\" (UniqueName: \"kubernetes.io/projected/f076b615-76d7-4867-ae84-0374653e85eb-kube-api-access-vspxs\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.968917 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.968888    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctlrq\" (UniqueName: \"kubernetes.io/projected/ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a-kube-api-access-ctlrq\") pod \"openshift-state-metrics-5669946b84-cjq55\" (UID: \"ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"
Apr 16 16:30:35.971785 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.971741    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f076b615-76d7-4867-ae84-0374653e85eb-node-exporter-textfile\") pod \"node-exporter-s6mx4\" (UID: \"f076b615-76d7-4867-ae84-0374653e85eb\") " pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:35.997724 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:35.997704    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"
Apr 16 16:30:36.035643 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:36.035619    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s6mx4"
Apr 16 16:30:38.774120 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.774089    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6947b544f5-f4vb2"]
Apr 16 16:30:38.793579 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.793536    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6947b544f5-f4vb2"]
Apr 16 16:30:38.793741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.793723    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:38.796254 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.796220    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Apr 16 16:30:38.796354 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.796333    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-aek61dfl6t3pu\""
Apr 16 16:30:38.796416 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.796220    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Apr 16 16:30:38.796726 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.796702    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Apr 16 16:30:38.796726 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.796723    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Apr 16 16:30:38.796885 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.796725    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-h479x\""
Apr 16 16:30:38.796885 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.796801    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Apr 16 16:30:38.985787 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.985750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:38.985966 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.985796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:38.985966 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.985821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf5gb\" (UniqueName: \"kubernetes.io/projected/351052b4-54f5-459b-8b1e-4542e7b6d41d-kube-api-access-kf5gb\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:38.985966 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.985909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-thanos-querier-tls\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:38.986197 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.985971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-grpc-tls\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:38.986197 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.986056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:38.986197 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.986104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/351052b4-54f5-459b-8b1e-4542e7b6d41d-metrics-client-ca\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:38.986197 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:38.986127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.086598 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.086504    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-grpc-tls\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.086739 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.086613    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.086739 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.086661    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/351052b4-54f5-459b-8b1e-4542e7b6d41d-metrics-client-ca\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.086739 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.086691    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.087005 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.086976    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.087147 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.087020    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.087147 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.087048    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kf5gb\" (UniqueName: \"kubernetes.io/projected/351052b4-54f5-459b-8b1e-4542e7b6d41d-kube-api-access-kf5gb\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.087147 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.087079    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-thanos-querier-tls\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.087427 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.087391    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/351052b4-54f5-459b-8b1e-4542e7b6d41d-metrics-client-ca\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.089859 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.089805    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.089859 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.089823    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-grpc-tls\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.090231 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.090207    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.090317 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.090297    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-thanos-querier-tls\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.090404 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.090344    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.090471 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.090438    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/351052b4-54f5-459b-8b1e-4542e7b6d41d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.095086 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.095064    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf5gb\" (UniqueName: \"kubernetes.io/projected/351052b4-54f5-459b-8b1e-4542e7b6d41d-kube-api-access-kf5gb\") pod \"thanos-querier-6947b544f5-f4vb2\" (UID: \"351052b4-54f5-459b-8b1e-4542e7b6d41d\") " pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:39.105850 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:39.105828    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:41.508918 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.508877    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jrfnm"
Apr 16 16:30:41.853814 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.853743    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-644dcd4585-bnnh8"]
Apr 16 16:30:41.893119 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.893016    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 16 16:30:41.893119 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.893092    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:41.910858 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.910833    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Apr 16 16:30:41.912215 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.912177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-oauth-config\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:41.912340 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.912216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-config\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:41.912340 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.912249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-service-ca\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:41.912340 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.912328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-serving-cert\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:41.912506 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.912351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-oauth-serving-cert\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:41.912506 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.912386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-trusted-ca-bundle\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:41.912506 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.912420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwc2s\" (UniqueName: \"kubernetes.io/projected/38ce9b1e-ae3d-48f8-85b2-9953825acddf-kube-api-access-dwc2s\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:41.914301 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.914283    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-644dcd4585-bnnh8"]
Apr 16 16:30:41.914301 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.914303    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 16 16:30:41.914451 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.914431    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:41.917074 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.917002    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Apr 16 16:30:41.917271 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.917075    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Apr 16 16:30:41.917271 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.917160    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Apr 16 16:30:41.917464 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.917342    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Apr 16 16:30:41.919736 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.919299    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Apr 16 16:30:41.919736 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.919319    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-auu6g6r71put3\""
Apr 16 16:30:41.919736 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.919366    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Apr 16 16:30:41.919736 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.919495    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hcgqb\""
Apr 16 16:30:41.919736 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.919615    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Apr 16 16:30:41.920050 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.920033    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Apr 16 16:30:41.920232 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.920217    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Apr 16 16:30:41.920429 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.920409    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Apr 16 16:30:41.920501 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.920463    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Apr 16 16:30:41.920623 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.920515    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Apr 16 16:30:41.924935 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:41.924914    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Apr 16 16:30:42.012820 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.012788    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dwc2s\" (UniqueName: \"kubernetes.io/projected/38ce9b1e-ae3d-48f8-85b2-9953825acddf-kube-api-access-dwc2s\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:42.013028 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.012838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013028 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.012865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013028 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.012903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013028 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.012932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013028 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.012957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13a73e5c-19b8-43cd-9ad0-3db3153f7816-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013028 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.012999    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-oauth-config\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:42.013028 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013028    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-config\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:42.013409 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-web-config\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013409 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013409 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013409 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013135    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-service-ca\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:42.013409 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-config\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013409 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013201    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013409 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013409 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13a73e5c-19b8-43cd-9ad0-3db3153f7816-config-out\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013409 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013409 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013339    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013409 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013409 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013397    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-serving-cert\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:42.013999 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013422    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-oauth-serving-cert\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:42.013999 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013999 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.013999 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013835    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-config\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:42.013999 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.013934    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-trusted-ca-bundle\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:42.014237 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.014070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9gfl\" (UniqueName: \"kubernetes.io/projected/13a73e5c-19b8-43cd-9ad0-3db3153f7816-kube-api-access-w9gfl\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.014237 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.014024    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-service-ca\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:42.015692 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.014920    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-oauth-serving-cert\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:42.015692 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.015637    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-trusted-ca-bundle\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:42.018218 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.018159    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-oauth-config\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:42.018305 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.018242    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-serving-cert\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:42.023452 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.023433    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwc2s\" (UniqueName: \"kubernetes.io/projected/38ce9b1e-ae3d-48f8-85b2-9953825acddf-kube-api-access-dwc2s\") pod \"console-644dcd4585-bnnh8\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") " pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:42.115683 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115414    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.115683 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115466    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.115683 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115500    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.115683 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115535    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.115683 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115590    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.115683 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115618    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w9gfl\" (UniqueName: \"kubernetes.io/projected/13a73e5c-19b8-43cd-9ad0-3db3153f7816-kube-api-access-w9gfl\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.115683 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115655    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.115683 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115678    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.117083 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115714    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.117083 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115745    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.117083 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115781    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13a73e5c-19b8-43cd-9ad0-3db3153f7816-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.117083 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115821    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-web-config\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.117083 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115837    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.117083 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115857    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.117083 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115878    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-config\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.117083 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115901    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.117083 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115933    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.117083 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.115964    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13a73e5c-19b8-43cd-9ad0-3db3153f7816-config-out\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.120275 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.117878    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.120275 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.119769    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.122382 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.121159    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.122382 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.122263    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.122587 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.122544    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.123404 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.123111    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.123404 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.123250    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.123404 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.123302    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13a73e5c-19b8-43cd-9ad0-3db3153f7816-config-out\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.123404 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.123361    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-web-config\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.123700 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.123496    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.124267 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.123946    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-config\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.124267 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.124027    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.124267 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.124190    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.124267 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.124228    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.125938 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.125916    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9gfl\" (UniqueName: \"kubernetes.io/projected/13a73e5c-19b8-43cd-9ad0-3db3153f7816-kube-api-access-w9gfl\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.126161 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.126145    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13a73e5c-19b8-43cd-9ad0-3db3153f7816-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.126368 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.126345    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.142069 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.142017    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:42.206915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.206886    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:42.225932 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:42.225909    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:44.375077 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.375029    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6947b544f5-f4vb2"]
Apr 16 16:30:44.376691 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:30:44.376666    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351052b4_54f5_459b_8b1e_4542e7b6d41d.slice/crio-38398ab88ed196d47e963b7582b205e960be210b1c151d1918728ef7cbdc4e71 WatchSource:0}: Error finding container 38398ab88ed196d47e963b7582b205e960be210b1c151d1918728ef7cbdc4e71: Status 404 returned error can't find the container with id 38398ab88ed196d47e963b7582b205e960be210b1c151d1918728ef7cbdc4e71
Apr 16 16:30:44.411066 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.411031    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-644dcd4585-bnnh8"]
Apr 16 16:30:44.413646 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:30:44.413621    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ce9b1e_ae3d_48f8_85b2_9953825acddf.slice/crio-31eddd48c38d4a493c365aa48fdbee2d03ccf620abd00f0550a6c2a40544a0e6 WatchSource:0}: Error finding container 31eddd48c38d4a493c365aa48fdbee2d03ccf620abd00f0550a6c2a40544a0e6: Status 404 returned error can't find the container with id 31eddd48c38d4a493c365aa48fdbee2d03ccf620abd00f0550a6c2a40544a0e6
Apr 16 16:30:44.547812 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.547784    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-586b57c7b4-fxvzp" event={"ID":"b99ebca1-5a6b-44b9-8db2-e29fad719f8b","Type":"ContainerStarted","Data":"583731d0a616b809f780ed1d43396ace23b332facd8ae49d9028893d31da3dd2"}
Apr 16 16:30:44.548036 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.548011    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-586b57c7b4-fxvzp"
Apr 16 16:30:44.549302 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.549277    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4b67854d-nv7hr" event={"ID":"cde4edfa-4c1f-4006-8f70-12a437387527","Type":"ContainerStarted","Data":"acfa17e71df4f92d21e9258749551b02817c0b67cdfbfcc0d7ee3edd16aa0f3a"}
Apr 16 16:30:44.549417 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.549278    2578 patch_prober.go:28] interesting pod/downloads-586b57c7b4-fxvzp container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.8:8080/\": dial tcp 10.134.0.8:8080: connect: connection refused" start-of-body=
Apr 16 16:30:44.549417 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.549342    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-586b57c7b4-fxvzp" podUID="b99ebca1-5a6b-44b9-8db2-e29fad719f8b" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.8:8080/\": dial tcp 10.134.0.8:8080: connect: connection refused"
Apr 16 16:30:44.550826 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.550800    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6mx4" event={"ID":"f076b615-76d7-4867-ae84-0374653e85eb","Type":"ContainerStarted","Data":"97dff6bdca703ec26b080e50025796140ad249f4089c783693765f5a068354aa"}
Apr 16 16:30:44.552038 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.552014    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644dcd4585-bnnh8" event={"ID":"38ce9b1e-ae3d-48f8-85b2-9953825acddf","Type":"ContainerStarted","Data":"0224c0c517187f5ee1f928247d2d44252f81c0686a1364a345f5ff7c74100424"}
Apr 16 16:30:44.552138 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.552041    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644dcd4585-bnnh8" event={"ID":"38ce9b1e-ae3d-48f8-85b2-9953825acddf","Type":"ContainerStarted","Data":"31eddd48c38d4a493c365aa48fdbee2d03ccf620abd00f0550a6c2a40544a0e6"}
Apr 16 16:30:44.553102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.553082    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2" event={"ID":"351052b4-54f5-459b-8b1e-4542e7b6d41d","Type":"ContainerStarted","Data":"38398ab88ed196d47e963b7582b205e960be210b1c151d1918728ef7cbdc4e71"}
Apr 16 16:30:44.568857 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.568816    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-586b57c7b4-fxvzp" podStartSLOduration=2.314931503 podStartE2EDuration="20.568803427s" podCreationTimestamp="2026-04-16 16:30:24 +0000 UTC" firstStartedPulling="2026-04-16 16:30:25.979164948 +0000 UTC m=+35.376381314" lastFinishedPulling="2026-04-16 16:30:44.233036854 +0000 UTC m=+53.630253238" observedRunningTime="2026-04-16 16:30:44.567691804 +0000 UTC m=+53.964908193" watchObservedRunningTime="2026-04-16 16:30:44.568803427 +0000 UTC m=+53.966019815"
Apr 16 16:30:44.577140 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.577117    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:44.577251 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.577153    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:44.582128 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.582106    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:44.584126 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.584093    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7c4b67854d-nv7hr" podStartSLOduration=1.077362709 podStartE2EDuration="10.584082436s" podCreationTimestamp="2026-04-16 16:30:34 +0000 UTC" firstStartedPulling="2026-04-16 16:30:34.710917898 +0000 UTC m=+44.108134264" lastFinishedPulling="2026-04-16 16:30:44.217637624 +0000 UTC m=+53.614853991" observedRunningTime="2026-04-16 16:30:44.583724553 +0000 UTC m=+53.980940952" watchObservedRunningTime="2026-04-16 16:30:44.584082436 +0000 UTC m=+53.981298825"
Apr 16 16:30:44.588434 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.588414    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-5669946b84-cjq55"]
Apr 16 16:30:44.593593 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.593543    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 16 16:30:44.597043 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:30:44.597019    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a73e5c_19b8_43cd_9ad0_3db3153f7816.slice/crio-516339e8f09564f372a19c7fdcd177a318b868f704f0485e0302e3abe0aa0691 WatchSource:0}: Error finding container 516339e8f09564f372a19c7fdcd177a318b868f704f0485e0302e3abe0aa0691: Status 404 returned error can't find the container with id 516339e8f09564f372a19c7fdcd177a318b868f704f0485e0302e3abe0aa0691
Apr 16 16:30:44.601917 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:44.601882    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-644dcd4585-bnnh8" podStartSLOduration=3.6018709270000002 podStartE2EDuration="3.601870927s" podCreationTimestamp="2026-04-16 16:30:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:30:44.600985171 +0000 UTC m=+53.998201559" watchObservedRunningTime="2026-04-16 16:30:44.601870927 +0000 UTC m=+53.999087312"
Apr 16 16:30:45.559617 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:45.559528    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13a73e5c-19b8-43cd-9ad0-3db3153f7816","Type":"ContainerStarted","Data":"516339e8f09564f372a19c7fdcd177a318b868f704f0485e0302e3abe0aa0691"}
Apr 16 16:30:45.562465 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:45.562388    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55" event={"ID":"ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a","Type":"ContainerStarted","Data":"c9b8ebd29d3768de4b20b71f8395a42f0abd2099c44d90b35778dcb2e901dbd3"}
Apr 16 16:30:45.562465 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:45.562422    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55" event={"ID":"ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a","Type":"ContainerStarted","Data":"195152af3be26945534024d7e0d015ed3338d3308c98bf3ba9c9df7ee6405b88"}
Apr 16 16:30:45.562465 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:45.562437    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55" event={"ID":"ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a","Type":"ContainerStarted","Data":"3aa2b4f447831011a323c289a43eefdd2aabb136e4f87e3cb216c8cb7e5677cf"}
Apr 16 16:30:45.581872 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:45.581852    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:30:45.591137 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:45.591115    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-586b57c7b4-fxvzp"
Apr 16 16:30:46.568727 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:46.568686    2578 generic.go:358] "Generic (PLEG): container finished" podID="f076b615-76d7-4867-ae84-0374653e85eb" containerID="012b96c70d5a3458eed7d86176ebe8771bb3a75bb76dfa385d4f51e62604b0eb" exitCode=0
Apr 16 16:30:46.569392 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:46.568816    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6mx4" event={"ID":"f076b615-76d7-4867-ae84-0374653e85eb","Type":"ContainerDied","Data":"012b96c70d5a3458eed7d86176ebe8771bb3a75bb76dfa385d4f51e62604b0eb"}
Apr 16 16:30:48.454328 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:48.454300    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wd8ld"
Apr 16 16:30:48.580329 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:48.580291    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6mx4" event={"ID":"f076b615-76d7-4867-ae84-0374653e85eb","Type":"ContainerStarted","Data":"3b65a85b8d44b21970f020202a95c4b8cbcee88e2c16b5734b8991abf3f86c02"}
Apr 16 16:30:48.580452 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:48.580337    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6mx4" event={"ID":"f076b615-76d7-4867-ae84-0374653e85eb","Type":"ContainerStarted","Data":"e7cec75b4d0a5874cef38c53c7afb7594e0e095d1d7d03680c384967b63fffc2"}
Apr 16 16:30:48.582607 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:48.582464    2578 generic.go:358] "Generic (PLEG): container finished" podID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerID="07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5" exitCode=0
Apr 16 16:30:48.582607 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:48.582549    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13a73e5c-19b8-43cd-9ad0-3db3153f7816","Type":"ContainerDied","Data":"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5"}
Apr 16 16:30:48.585753 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:48.585715    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2" event={"ID":"351052b4-54f5-459b-8b1e-4542e7b6d41d","Type":"ContainerStarted","Data":"4b8f6f780f7911d20340bbe81eb70df28b89e34dbb4f0c932b323e25ef0c8ff3"}
Apr 16 16:30:48.585916 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:48.585896    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2" event={"ID":"351052b4-54f5-459b-8b1e-4542e7b6d41d","Type":"ContainerStarted","Data":"9b5e57940c29c1b257d79a25a9ef40116e3049f3580f9faffe50aac8f0bd84ca"}
Apr 16 16:30:48.586038 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:48.586023    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2" event={"ID":"351052b4-54f5-459b-8b1e-4542e7b6d41d","Type":"ContainerStarted","Data":"038cf17884ae93478fd99e71e7d2922ec447cdfb490b3f56877af8673b46786a"}
Apr 16 16:30:48.588047 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:48.588022    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55" event={"ID":"ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a","Type":"ContainerStarted","Data":"22a413290f9551ec03492480de7db467aa02b88046504f96ca16a0189404f6dc"}
Apr 16 16:30:48.600329 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:48.600285    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s6mx4" podStartSLOduration=12.264642124 podStartE2EDuration="13.600271483s" podCreationTimestamp="2026-04-16 16:30:35 +0000 UTC" firstStartedPulling="2026-04-16 16:30:44.220678087 +0000 UTC m=+53.617894453" lastFinishedPulling="2026-04-16 16:30:45.556307439 +0000 UTC m=+54.953523812" observedRunningTime="2026-04-16 16:30:48.598809438 +0000 UTC m=+57.996025827" watchObservedRunningTime="2026-04-16 16:30:48.600271483 +0000 UTC m=+57.997487870"
Apr 16 16:30:48.634680 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:48.634543    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-5669946b84-cjq55" podStartSLOduration=10.260773644 podStartE2EDuration="13.634529649s" podCreationTimestamp="2026-04-16 16:30:35 +0000 UTC" firstStartedPulling="2026-04-16 16:30:44.797232675 +0000 UTC m=+54.194449058" lastFinishedPulling="2026-04-16 16:30:48.170988692 +0000 UTC m=+57.568205063" observedRunningTime="2026-04-16 16:30:48.633945677 +0000 UTC m=+58.031162065" watchObservedRunningTime="2026-04-16 16:30:48.634529649 +0000 UTC m=+58.031746038"
Apr 16 16:30:50.600939 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:50.600857    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2" event={"ID":"351052b4-54f5-459b-8b1e-4542e7b6d41d","Type":"ContainerStarted","Data":"2853a049c4d6b9cf8ab6a03e415f4ef1d6997fb9af6ac8f2f777e8443acf45e7"}
Apr 16 16:30:50.601388 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:50.601139    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:50.601388 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:50.601166    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2" event={"ID":"351052b4-54f5-459b-8b1e-4542e7b6d41d","Type":"ContainerStarted","Data":"91f91ec67628682f89f49ba76ae2728aaff5bb14e51256c2821855f33d046a63"}
Apr 16 16:30:50.601388 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:50.601198    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2" event={"ID":"351052b4-54f5-459b-8b1e-4542e7b6d41d","Type":"ContainerStarted","Data":"66fa694399f7e9dba25bd9528430ca3081863ca38759b6b4acdbc8fd980e21a0"}
Apr 16 16:30:50.626949 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:50.626890    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2" podStartSLOduration=7.095068135 podStartE2EDuration="12.626871606s" podCreationTimestamp="2026-04-16 16:30:38 +0000 UTC" firstStartedPulling="2026-04-16 16:30:44.379621153 +0000 UTC m=+53.776837536" lastFinishedPulling="2026-04-16 16:30:49.911424626 +0000 UTC m=+59.308641007" observedRunningTime="2026-04-16 16:30:50.624773378 +0000 UTC m=+60.021989768" watchObservedRunningTime="2026-04-16 16:30:50.626871606 +0000 UTC m=+60.024087998"
Apr 16 16:30:52.208717 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:52.208681    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:52.209668 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:52.209525    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:52.214980 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:52.214958    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:52.611224 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:52.611166    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13a73e5c-19b8-43cd-9ad0-3db3153f7816","Type":"ContainerStarted","Data":"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44"}
Apr 16 16:30:52.611224 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:52.611211    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13a73e5c-19b8-43cd-9ad0-3db3153f7816","Type":"ContainerStarted","Data":"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af"}
Apr 16 16:30:52.616710 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:52.616684    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:30:52.667401 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:52.667210    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c4b67854d-nv7hr"]
Apr 16 16:30:53.617933 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:53.617893    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13a73e5c-19b8-43cd-9ad0-3db3153f7816","Type":"ContainerStarted","Data":"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1"}
Apr 16 16:30:53.617933 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:53.617936    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13a73e5c-19b8-43cd-9ad0-3db3153f7816","Type":"ContainerStarted","Data":"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10"}
Apr 16 16:30:53.618511 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:53.617952    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13a73e5c-19b8-43cd-9ad0-3db3153f7816","Type":"ContainerStarted","Data":"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7"}
Apr 16 16:30:53.618511 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:53.617964    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13a73e5c-19b8-43cd-9ad0-3db3153f7816","Type":"ContainerStarted","Data":"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3"}
Apr 16 16:30:53.645269 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:53.645215    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.960468743 podStartE2EDuration="12.645195876s" podCreationTimestamp="2026-04-16 16:30:41 +0000 UTC" firstStartedPulling="2026-04-16 16:30:44.598910531 +0000 UTC m=+53.996126897" lastFinishedPulling="2026-04-16 16:30:52.283637644 +0000 UTC m=+61.680854030" observedRunningTime="2026-04-16 16:30:53.642637125 +0000 UTC m=+63.039853515" watchObservedRunningTime="2026-04-16 16:30:53.645195876 +0000 UTC m=+63.042412265"
Apr 16 16:30:55.946200 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:55.946160    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs\") pod \"network-metrics-daemon-wq65n\" (UID: \"21a9695d-812c-4ad6-9b18-ade579b343e0\") " pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:55.948433 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:55.948415    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Apr 16 16:30:55.959230 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:55.959205    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a9695d-812c-4ad6-9b18-ade579b343e0-metrics-certs\") pod \"network-metrics-daemon-wq65n\" (UID: \"21a9695d-812c-4ad6-9b18-ade579b343e0\") " pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:56.047281 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:56.047252    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5wx\" (UniqueName: \"kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx\") pod \"network-check-target-w2mqw\" (UID: \"29bd11bf-6683-4b59-9b71-e5f556a0ad04\") " pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:56.049563 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:56.049534    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Apr 16 16:30:56.060232 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:56.060214    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Apr 16 16:30:56.071267 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:56.071241    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf5wx\" (UniqueName: \"kubernetes.io/projected/29bd11bf-6683-4b59-9b71-e5f556a0ad04-kube-api-access-zf5wx\") pod \"network-check-target-w2mqw\" (UID: \"29bd11bf-6683-4b59-9b71-e5f556a0ad04\") " pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:56.145782 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:56.145761    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tqp8k\""
Apr 16 16:30:56.152056 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:56.152038    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ws7rw\""
Apr 16 16:30:56.153782 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:56.153763    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wq65n"
Apr 16 16:30:56.160951 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:56.160937    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:30:56.286130 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:56.286102    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w2mqw"]
Apr 16 16:30:56.290306 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:30:56.290282    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bd11bf_6683_4b59_9b71_e5f556a0ad04.slice/crio-67f8102f5eb4c8e5819ec4aba724b67abbc290d7bf8c0490922ca8c1080ddffd WatchSource:0}: Error finding container 67f8102f5eb4c8e5819ec4aba724b67abbc290d7bf8c0490922ca8c1080ddffd: Status 404 returned error can't find the container with id 67f8102f5eb4c8e5819ec4aba724b67abbc290d7bf8c0490922ca8c1080ddffd
Apr 16 16:30:56.306657 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:56.306635    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wq65n"]
Apr 16 16:30:56.315885 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:30:56.315861    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a9695d_812c_4ad6_9b18_ade579b343e0.slice/crio-c518d082102ac9f79d3331787117a1126e0eeb7b9a68067a29c85a694a2ba40b WatchSource:0}: Error finding container c518d082102ac9f79d3331787117a1126e0eeb7b9a68067a29c85a694a2ba40b: Status 404 returned error can't find the container with id c518d082102ac9f79d3331787117a1126e0eeb7b9a68067a29c85a694a2ba40b
Apr 16 16:30:56.611202 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:56.611145    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6947b544f5-f4vb2"
Apr 16 16:30:56.628190 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:56.628168    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wq65n" event={"ID":"21a9695d-812c-4ad6-9b18-ade579b343e0","Type":"ContainerStarted","Data":"c518d082102ac9f79d3331787117a1126e0eeb7b9a68067a29c85a694a2ba40b"}
Apr 16 16:30:56.629190 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:56.629159    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w2mqw" event={"ID":"29bd11bf-6683-4b59-9b71-e5f556a0ad04","Type":"ContainerStarted","Data":"67f8102f5eb4c8e5819ec4aba724b67abbc290d7bf8c0490922ca8c1080ddffd"}
Apr 16 16:30:57.226902 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:57.226867    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:30:58.638796 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:58.638693    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wq65n" event={"ID":"21a9695d-812c-4ad6-9b18-ade579b343e0","Type":"ContainerStarted","Data":"71cfdd18ed5bfa1dfb98a752d9d6a444e49973a13499c642b3b13392c375bea4"}
Apr 16 16:30:58.638796 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:58.638753    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wq65n" event={"ID":"21a9695d-812c-4ad6-9b18-ade579b343e0","Type":"ContainerStarted","Data":"9dfd18c1877828f52d7b3c1a61ccfe80da63dfe39d9bdaed448a383f3f9d99f6"}
Apr 16 16:30:58.653790 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:30:58.653735    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wq65n" podStartSLOduration=66.018790949 podStartE2EDuration="1m7.653717386s" podCreationTimestamp="2026-04-16 16:29:51 +0000 UTC" firstStartedPulling="2026-04-16 16:30:56.317650187 +0000 UTC m=+65.714866553" lastFinishedPulling="2026-04-16 16:30:57.952576622 +0000 UTC m=+67.349792990" observedRunningTime="2026-04-16 16:30:58.653519595 +0000 UTC m=+68.050735984" watchObservedRunningTime="2026-04-16 16:30:58.653717386 +0000 UTC m=+68.050933776"
Apr 16 16:31:00.646572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:00.646516    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w2mqw" event={"ID":"29bd11bf-6683-4b59-9b71-e5f556a0ad04","Type":"ContainerStarted","Data":"bab9bfee1a279b9872b8a3a6e8878b093bd1539e27d0399a608e1ea08365438a"}
Apr 16 16:31:00.662144 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:00.662099    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w2mqw" podStartSLOduration=66.213055007 podStartE2EDuration="1m9.662085155s" podCreationTimestamp="2026-04-16 16:29:51 +0000 UTC" firstStartedPulling="2026-04-16 16:30:56.292661032 +0000 UTC m=+65.689877399" lastFinishedPulling="2026-04-16 16:30:59.741691166 +0000 UTC m=+69.138907547" observedRunningTime="2026-04-16 16:31:00.660619327 +0000 UTC m=+70.057835716" watchObservedRunningTime="2026-04-16 16:31:00.662085155 +0000 UTC m=+70.059301575"
Apr 16 16:31:01.649716 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:01.649685    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:31:17.713756 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:17.713696    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-7c4b67854d-nv7hr" podUID="cde4edfa-4c1f-4006-8f70-12a437387527" containerName="console" containerID="cri-o://acfa17e71df4f92d21e9258749551b02817c0b67cdfbfcc0d7ee3edd16aa0f3a" gracePeriod=15
Apr 16 16:31:17.969115 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:17.969064    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c4b67854d-nv7hr_cde4edfa-4c1f-4006-8f70-12a437387527/console/0.log"
Apr 16 16:31:17.969216 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:17.969121    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:31:18.114725 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.114697    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-console-config\") pod \"cde4edfa-4c1f-4006-8f70-12a437387527\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") "
Apr 16 16:31:18.114873 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.114770    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cde4edfa-4c1f-4006-8f70-12a437387527-console-serving-cert\") pod \"cde4edfa-4c1f-4006-8f70-12a437387527\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") "
Apr 16 16:31:18.114873 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.114794    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cde4edfa-4c1f-4006-8f70-12a437387527-console-oauth-config\") pod \"cde4edfa-4c1f-4006-8f70-12a437387527\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") "
Apr 16 16:31:18.114873 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.114829    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlrx4\" (UniqueName: \"kubernetes.io/projected/cde4edfa-4c1f-4006-8f70-12a437387527-kube-api-access-hlrx4\") pod \"cde4edfa-4c1f-4006-8f70-12a437387527\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") "
Apr 16 16:31:18.114873 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.114852    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-service-ca\") pod \"cde4edfa-4c1f-4006-8f70-12a437387527\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") "
Apr 16 16:31:18.115089 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.114914    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-oauth-serving-cert\") pod \"cde4edfa-4c1f-4006-8f70-12a437387527\" (UID: \"cde4edfa-4c1f-4006-8f70-12a437387527\") "
Apr 16 16:31:18.115089 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.114911    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-console-config" (OuterVolumeSpecName: "console-config") pod "cde4edfa-4c1f-4006-8f70-12a437387527" (UID: "cde4edfa-4c1f-4006-8f70-12a437387527"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:31:18.115190 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.115168    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-console-config\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:18.115252 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.115240    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-service-ca" (OuterVolumeSpecName: "service-ca") pod "cde4edfa-4c1f-4006-8f70-12a437387527" (UID: "cde4edfa-4c1f-4006-8f70-12a437387527"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:31:18.115303 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.115271    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cde4edfa-4c1f-4006-8f70-12a437387527" (UID: "cde4edfa-4c1f-4006-8f70-12a437387527"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:31:18.116987 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.116949    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde4edfa-4c1f-4006-8f70-12a437387527-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cde4edfa-4c1f-4006-8f70-12a437387527" (UID: "cde4edfa-4c1f-4006-8f70-12a437387527"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:31:18.117089 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.117038    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde4edfa-4c1f-4006-8f70-12a437387527-kube-api-access-hlrx4" (OuterVolumeSpecName: "kube-api-access-hlrx4") pod "cde4edfa-4c1f-4006-8f70-12a437387527" (UID: "cde4edfa-4c1f-4006-8f70-12a437387527"). InnerVolumeSpecName "kube-api-access-hlrx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:31:18.117156 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.117140    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde4edfa-4c1f-4006-8f70-12a437387527-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cde4edfa-4c1f-4006-8f70-12a437387527" (UID: "cde4edfa-4c1f-4006-8f70-12a437387527"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:31:18.215748 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.215724    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlrx4\" (UniqueName: \"kubernetes.io/projected/cde4edfa-4c1f-4006-8f70-12a437387527-kube-api-access-hlrx4\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:18.215748 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.215746    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-service-ca\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:18.215898 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.215757    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cde4edfa-4c1f-4006-8f70-12a437387527-oauth-serving-cert\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:18.215898 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.215766    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cde4edfa-4c1f-4006-8f70-12a437387527-console-serving-cert\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:18.215898 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.215774    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cde4edfa-4c1f-4006-8f70-12a437387527-console-oauth-config\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:18.697496 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.697471    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c4b67854d-nv7hr_cde4edfa-4c1f-4006-8f70-12a437387527/console/0.log"
Apr 16 16:31:18.697661 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.697509    2578 generic.go:358] "Generic (PLEG): container finished" podID="cde4edfa-4c1f-4006-8f70-12a437387527" containerID="acfa17e71df4f92d21e9258749551b02817c0b67cdfbfcc0d7ee3edd16aa0f3a" exitCode=2
Apr 16 16:31:18.697661 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.697538    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4b67854d-nv7hr" event={"ID":"cde4edfa-4c1f-4006-8f70-12a437387527","Type":"ContainerDied","Data":"acfa17e71df4f92d21e9258749551b02817c0b67cdfbfcc0d7ee3edd16aa0f3a"}
Apr 16 16:31:18.697661 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.697594    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4b67854d-nv7hr" event={"ID":"cde4edfa-4c1f-4006-8f70-12a437387527","Type":"ContainerDied","Data":"52705f0816b95fd60a6355f79a4353d8d5178dc0665991043a384857c412dbb3"}
Apr 16 16:31:18.697661 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.697598    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4b67854d-nv7hr"
Apr 16 16:31:18.697661 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.697608    2578 scope.go:117] "RemoveContainer" containerID="acfa17e71df4f92d21e9258749551b02817c0b67cdfbfcc0d7ee3edd16aa0f3a"
Apr 16 16:31:18.705468 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.705449    2578 scope.go:117] "RemoveContainer" containerID="acfa17e71df4f92d21e9258749551b02817c0b67cdfbfcc0d7ee3edd16aa0f3a"
Apr 16 16:31:18.705761 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:31:18.705731    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acfa17e71df4f92d21e9258749551b02817c0b67cdfbfcc0d7ee3edd16aa0f3a\": container with ID starting with acfa17e71df4f92d21e9258749551b02817c0b67cdfbfcc0d7ee3edd16aa0f3a not found: ID does not exist" containerID="acfa17e71df4f92d21e9258749551b02817c0b67cdfbfcc0d7ee3edd16aa0f3a"
Apr 16 16:31:18.705834 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.705764    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acfa17e71df4f92d21e9258749551b02817c0b67cdfbfcc0d7ee3edd16aa0f3a"} err="failed to get container status \"acfa17e71df4f92d21e9258749551b02817c0b67cdfbfcc0d7ee3edd16aa0f3a\": rpc error: code = NotFound desc = could not find container \"acfa17e71df4f92d21e9258749551b02817c0b67cdfbfcc0d7ee3edd16aa0f3a\": container with ID starting with acfa17e71df4f92d21e9258749551b02817c0b67cdfbfcc0d7ee3edd16aa0f3a not found: ID does not exist"
Apr 16 16:31:18.716881 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.716861    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c4b67854d-nv7hr"]
Apr 16 16:31:18.721843 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:18.721822    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c4b67854d-nv7hr"]
Apr 16 16:31:19.234657 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:19.234629    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde4edfa-4c1f-4006-8f70-12a437387527" path="/var/lib/kubelet/pods/cde4edfa-4c1f-4006-8f70-12a437387527/volumes"
Apr 16 16:31:32.654457 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:32.654422    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w2mqw"
Apr 16 16:31:33.410754 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:33.410727    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:33.471344 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:33.471318    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:33.752445 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:33.752421    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.233992 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.233942    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 16 16:31:50.237836 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.237774    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="prometheus" containerID="cri-o://9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af" gracePeriod=600
Apr 16 16:31:50.239049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.238023    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="kube-rbac-proxy" containerID="cri-o://40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10" gracePeriod=600
Apr 16 16:31:50.239049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.238164    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="kube-rbac-proxy-web" containerID="cri-o://d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7" gracePeriod=600
Apr 16 16:31:50.239049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.238182    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="config-reloader" containerID="cri-o://e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44" gracePeriod=600
Apr 16 16:31:50.239049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.238160    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="kube-rbac-proxy-thanos" containerID="cri-o://68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1" gracePeriod=600
Apr 16 16:31:50.239049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.238267    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="thanos-sidecar" containerID="cri-o://1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3" gracePeriod=600
Apr 16 16:31:50.491918 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.491856    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.532404 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.532375    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-metrics-client-ca\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.532404 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.532408    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-tls\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.532684 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.532438    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-kubelet-serving-ca-bundle\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.532826 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.532799    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:31:50.532894 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.532793    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:31:50.532894 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.532830    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-serving-certs-ca-bundle\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.532894 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.532870    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-thanos-prometheus-http-client-file\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.533053 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.532902    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13a73e5c-19b8-43cd-9ad0-3db3153f7816-config-out\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.533053 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.532935    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-trusted-ca-bundle\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.533053 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.532964    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-web-config\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.533053 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.533011    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9gfl\" (UniqueName: \"kubernetes.io/projected/13a73e5c-19b8-43cd-9ad0-3db3153f7816-kube-api-access-w9gfl\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.533053 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.533038    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-config\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.533299 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.533067    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-metrics-client-certs\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.533299 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.533071    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:31:50.533299 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.533094    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-k8s-rulefiles-0\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.533299 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.533138    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-grpc-tls\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.533299 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.533165    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.533299 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.533206    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-k8s-db\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.533299 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.533233    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.533299 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.533263    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13a73e5c-19b8-43cd-9ad0-3db3153f7816-tls-assets\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.533299 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.533289    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-kube-rbac-proxy\") pod \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\" (UID: \"13a73e5c-19b8-43cd-9ad0-3db3153f7816\") "
Apr 16 16:31:50.533765 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.533513    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-metrics-client-ca\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.533765 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.533531    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.533765 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.533547    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.535280 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.534304    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:31:50.537186 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.537064    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:31:50.537186 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.537156    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:31:50.537378 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.537286    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:31:50.537950 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.537920    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:31:50.538328 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.538302    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a73e5c-19b8-43cd-9ad0-3db3153f7816-kube-api-access-w9gfl" (OuterVolumeSpecName: "kube-api-access-w9gfl") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "kube-api-access-w9gfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:31:50.538526 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.538490    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:31:50.539224 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.539183    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:31:50.539460 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.539434    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:31:50.539712 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.539679    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:31:50.539839 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.539692    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a73e5c-19b8-43cd-9ad0-3db3153f7816-config-out" (OuterVolumeSpecName: "config-out") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:31:50.540149 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.540095    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:31:50.540246 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.540225    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a73e5c-19b8-43cd-9ad0-3db3153f7816-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:31:50.541267 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.541234    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-config" (OuterVolumeSpecName: "config") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:31:50.548376 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.548354    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-web-config" (OuterVolumeSpecName: "web-config") pod "13a73e5c-19b8-43cd-9ad0-3db3153f7816" (UID: "13a73e5c-19b8-43cd-9ad0-3db3153f7816"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:31:50.633982 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.633958    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-k8s-db\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.633982 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.633979    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.634102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.633991    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13a73e5c-19b8-43cd-9ad0-3db3153f7816-tls-assets\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.634102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.634000    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-kube-rbac-proxy\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.634102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.634009    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.634102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.634018    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.634102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.634028    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13a73e5c-19b8-43cd-9ad0-3db3153f7816-config-out\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.634102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.634037    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.634102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.634046    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-web-config\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.634102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.634054    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9gfl\" (UniqueName: \"kubernetes.io/projected/13a73e5c-19b8-43cd-9ad0-3db3153f7816-kube-api-access-w9gfl\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.634102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.634062    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-config\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.634102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.634071    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-metrics-client-certs\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.634102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.634081    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13a73e5c-19b8-43cd-9ad0-3db3153f7816-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.634102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.634089    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-grpc-tls\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.634102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.634100    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13a73e5c-19b8-43cd-9ad0-3db3153f7816-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:31:50.791843 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.791778    2578 generic.go:358] "Generic (PLEG): container finished" podID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerID="68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1" exitCode=0
Apr 16 16:31:50.791843 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.791797    2578 generic.go:358] "Generic (PLEG): container finished" podID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerID="40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10" exitCode=0
Apr 16 16:31:50.791843 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.791803    2578 generic.go:358] "Generic (PLEG): container finished" podID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerID="d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7" exitCode=0
Apr 16 16:31:50.791843 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.791809    2578 generic.go:358] "Generic (PLEG): container finished" podID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerID="1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3" exitCode=0
Apr 16 16:31:50.791843 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.791814    2578 generic.go:358] "Generic (PLEG): container finished" podID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerID="e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44" exitCode=0
Apr 16 16:31:50.791843 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.791819    2578 generic.go:358] "Generic (PLEG): container finished" podID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerID="9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af" exitCode=0
Apr 16 16:31:50.792094 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.791876    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.792094 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.791868    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13a73e5c-19b8-43cd-9ad0-3db3153f7816","Type":"ContainerDied","Data":"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1"}
Apr 16 16:31:50.792094 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.791985    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13a73e5c-19b8-43cd-9ad0-3db3153f7816","Type":"ContainerDied","Data":"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10"}
Apr 16 16:31:50.792094 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.791999    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13a73e5c-19b8-43cd-9ad0-3db3153f7816","Type":"ContainerDied","Data":"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7"}
Apr 16 16:31:50.792094 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.792009    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13a73e5c-19b8-43cd-9ad0-3db3153f7816","Type":"ContainerDied","Data":"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3"}
Apr 16 16:31:50.792094 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.792018    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13a73e5c-19b8-43cd-9ad0-3db3153f7816","Type":"ContainerDied","Data":"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44"}
Apr 16 16:31:50.792094 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.792030    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13a73e5c-19b8-43cd-9ad0-3db3153f7816","Type":"ContainerDied","Data":"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af"}
Apr 16 16:31:50.792094 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.792043    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13a73e5c-19b8-43cd-9ad0-3db3153f7816","Type":"ContainerDied","Data":"516339e8f09564f372a19c7fdcd177a318b868f704f0485e0302e3abe0aa0691"}
Apr 16 16:31:50.792094 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.792055    2578 scope.go:117] "RemoveContainer" containerID="68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1"
Apr 16 16:31:50.799734 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.799716    2578 scope.go:117] "RemoveContainer" containerID="40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10"
Apr 16 16:31:50.806087 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.806071    2578 scope.go:117] "RemoveContainer" containerID="d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7"
Apr 16 16:31:50.812134 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.812120    2578 scope.go:117] "RemoveContainer" containerID="1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3"
Apr 16 16:31:50.814415 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.814393    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 16 16:31:50.818814 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.818793    2578 scope.go:117] "RemoveContainer" containerID="e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44"
Apr 16 16:31:50.819810 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.819794    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 16 16:31:50.824892 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.824875    2578 scope.go:117] "RemoveContainer" containerID="9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af"
Apr 16 16:31:50.830806 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.830789    2578 scope.go:117] "RemoveContainer" containerID="07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5"
Apr 16 16:31:50.836681 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.836664    2578 scope.go:117] "RemoveContainer" containerID="68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1"
Apr 16 16:31:50.836906 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:31:50.836888    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1\": container with ID starting with 68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1 not found: ID does not exist" containerID="68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1"
Apr 16 16:31:50.836962 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.836912    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1"} err="failed to get container status \"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1\": rpc error: code = NotFound desc = could not find container \"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1\": container with ID starting with 68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1 not found: ID does not exist"
Apr 16 16:31:50.836962 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.836929    2578 scope.go:117] "RemoveContainer" containerID="40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10"
Apr 16 16:31:50.837171 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:31:50.837152    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10\": container with ID starting with 40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10 not found: ID does not exist" containerID="40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10"
Apr 16 16:31:50.837209 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.837181    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10"} err="failed to get container status \"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10\": rpc error: code = NotFound desc = could not find container \"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10\": container with ID starting with 40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10 not found: ID does not exist"
Apr 16 16:31:50.837209 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.837199    2578 scope.go:117] "RemoveContainer" containerID="d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7"
Apr 16 16:31:50.837410 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:31:50.837393    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7\": container with ID starting with d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7 not found: ID does not exist" containerID="d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7"
Apr 16 16:31:50.837448 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.837415    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7"} err="failed to get container status \"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7\": rpc error: code = NotFound desc = could not find container \"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7\": container with ID starting with d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7 not found: ID does not exist"
Apr 16 16:31:50.837448 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.837429    2578 scope.go:117] "RemoveContainer" containerID="1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3"
Apr 16 16:31:50.837689 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:31:50.837670    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3\": container with ID starting with 1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3 not found: ID does not exist" containerID="1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3"
Apr 16 16:31:50.837744 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.837693    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3"} err="failed to get container status \"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3\": rpc error: code = NotFound desc = could not find container \"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3\": container with ID starting with 1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3 not found: ID does not exist"
Apr 16 16:31:50.837744 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.837707    2578 scope.go:117] "RemoveContainer" containerID="e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44"
Apr 16 16:31:50.837915 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:31:50.837902    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44\": container with ID starting with e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44 not found: ID does not exist" containerID="e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44"
Apr 16 16:31:50.837951 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.837919    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44"} err="failed to get container status \"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44\": rpc error: code = NotFound desc = could not find container \"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44\": container with ID starting with e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44 not found: ID does not exist"
Apr 16 16:31:50.837951 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.837933    2578 scope.go:117] "RemoveContainer" containerID="9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af"
Apr 16 16:31:50.838136 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:31:50.838121    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af\": container with ID starting with 9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af not found: ID does not exist" containerID="9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af"
Apr 16 16:31:50.838191 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.838138    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af"} err="failed to get container status \"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af\": rpc error: code = NotFound desc = could not find container \"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af\": container with ID starting with 9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af not found: ID does not exist"
Apr 16 16:31:50.838191 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.838150    2578 scope.go:117] "RemoveContainer" containerID="07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5"
Apr 16 16:31:50.838370 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:31:50.838353    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5\": container with ID starting with 07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5 not found: ID does not exist" containerID="07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5"
Apr 16 16:31:50.838424 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.838375    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5"} err="failed to get container status \"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5\": rpc error: code = NotFound desc = could not find container \"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5\": container with ID starting with 07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5 not found: ID does not exist"
Apr 16 16:31:50.838424 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.838389    2578 scope.go:117] "RemoveContainer" containerID="68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1"
Apr 16 16:31:50.838628 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.838606    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1"} err="failed to get container status \"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1\": rpc error: code = NotFound desc = could not find container \"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1\": container with ID starting with 68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1 not found: ID does not exist"
Apr 16 16:31:50.838679 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.838630    2578 scope.go:117] "RemoveContainer" containerID="40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10"
Apr 16 16:31:50.838842 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.838827    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10"} err="failed to get container status \"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10\": rpc error: code = NotFound desc = could not find container \"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10\": container with ID starting with 40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10 not found: ID does not exist"
Apr 16 16:31:50.838891 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.838843    2578 scope.go:117] "RemoveContainer" containerID="d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7"
Apr 16 16:31:50.839076 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.839054    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7"} err="failed to get container status \"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7\": rpc error: code = NotFound desc = could not find container \"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7\": container with ID starting with d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7 not found: ID does not exist"
Apr 16 16:31:50.839151 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.839077    2578 scope.go:117] "RemoveContainer" containerID="1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3"
Apr 16 16:31:50.839294 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.839277    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3"} err="failed to get container status \"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3\": rpc error: code = NotFound desc = could not find container \"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3\": container with ID starting with 1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3 not found: ID does not exist"
Apr 16 16:31:50.839339 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.839294    2578 scope.go:117] "RemoveContainer" containerID="e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44"
Apr 16 16:31:50.839486 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.839469    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44"} err="failed to get container status \"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44\": rpc error: code = NotFound desc = could not find container \"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44\": container with ID starting with e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44 not found: ID does not exist"
Apr 16 16:31:50.839564 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.839487    2578 scope.go:117] "RemoveContainer" containerID="9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af"
Apr 16 16:31:50.839691 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.839676    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af"} err="failed to get container status \"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af\": rpc error: code = NotFound desc = could not find container \"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af\": container with ID starting with 9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af not found: ID does not exist"
Apr 16 16:31:50.839738 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.839692    2578 scope.go:117] "RemoveContainer" containerID="07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5"
Apr 16 16:31:50.839853 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.839838    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5"} err="failed to get container status \"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5\": rpc error: code = NotFound desc = could not find container \"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5\": container with ID starting with 07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5 not found: ID does not exist"
Apr 16 16:31:50.839889 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.839854    2578 scope.go:117] "RemoveContainer" containerID="68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1"
Apr 16 16:31:50.840020 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.840006    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1"} err="failed to get container status \"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1\": rpc error: code = NotFound desc = could not find container \"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1\": container with ID starting with 68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1 not found: ID does not exist"
Apr 16 16:31:50.840058 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.840020    2578 scope.go:117] "RemoveContainer" containerID="40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10"
Apr 16 16:31:50.840235 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.840196    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10"} err="failed to get container status \"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10\": rpc error: code = NotFound desc = could not find container \"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10\": container with ID starting with 40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10 not found: ID does not exist"
Apr 16 16:31:50.840235 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.840217    2578 scope.go:117] "RemoveContainer" containerID="d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7"
Apr 16 16:31:50.840504 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.840471    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7"} err="failed to get container status \"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7\": rpc error: code = NotFound desc = could not find container \"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7\": container with ID starting with d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7 not found: ID does not exist"
Apr 16 16:31:50.840675 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.840507    2578 scope.go:117] "RemoveContainer" containerID="1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3"
Apr 16 16:31:50.840870 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.840842    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3"} err="failed to get container status \"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3\": rpc error: code = NotFound desc = could not find container \"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3\": container with ID starting with 1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3 not found: ID does not exist"
Apr 16 16:31:50.840870 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.840869    2578 scope.go:117] "RemoveContainer" containerID="e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44"
Apr 16 16:31:50.841139 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.841115    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44"} err="failed to get container status \"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44\": rpc error: code = NotFound desc = could not find container \"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44\": container with ID starting with e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44 not found: ID does not exist"
Apr 16 16:31:50.841220 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.841140    2578 scope.go:117] "RemoveContainer" containerID="9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af"
Apr 16 16:31:50.841445 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.841422    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af"} err="failed to get container status \"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af\": rpc error: code = NotFound desc = could not find container \"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af\": container with ID starting with 9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af not found: ID does not exist"
Apr 16 16:31:50.841501 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.841448    2578 scope.go:117] "RemoveContainer" containerID="07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5"
Apr 16 16:31:50.841704 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.841685    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5"} err="failed to get container status \"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5\": rpc error: code = NotFound desc = could not find container \"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5\": container with ID starting with 07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5 not found: ID does not exist"
Apr 16 16:31:50.841771 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.841705    2578 scope.go:117] "RemoveContainer" containerID="68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1"
Apr 16 16:31:50.841955 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.841916    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1"} err="failed to get container status \"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1\": rpc error: code = NotFound desc = could not find container \"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1\": container with ID starting with 68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1 not found: ID does not exist"
Apr 16 16:31:50.842019 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.841956    2578 scope.go:117] "RemoveContainer" containerID="40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10"
Apr 16 16:31:50.842138 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842122    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 16 16:31:50.842178 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842164    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10"} err="failed to get container status \"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10\": rpc error: code = NotFound desc = could not find container \"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10\": container with ID starting with 40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10 not found: ID does not exist"
Apr 16 16:31:50.842223 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842181    2578 scope.go:117] "RemoveContainer" containerID="d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7"
Apr 16 16:31:50.842401 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842381    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7"} err="failed to get container status \"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7\": rpc error: code = NotFound desc = could not find container \"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7\": container with ID starting with d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7 not found: ID does not exist"
Apr 16 16:31:50.842460 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842402    2578 scope.go:117] "RemoveContainer" containerID="1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3"
Apr 16 16:31:50.842460 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842449    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="config-reloader"
Apr 16 16:31:50.842531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842462    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="config-reloader"
Apr 16 16:31:50.842531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842473    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="kube-rbac-proxy-thanos"
Apr 16 16:31:50.842531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842479    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="kube-rbac-proxy-thanos"
Apr 16 16:31:50.842531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842486    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="init-config-reloader"
Apr 16 16:31:50.842531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842493    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="init-config-reloader"
Apr 16 16:31:50.842531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842501    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="thanos-sidecar"
Apr 16 16:31:50.842531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842507    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="thanos-sidecar"
Apr 16 16:31:50.842531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842517    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="prometheus"
Apr 16 16:31:50.842531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842525    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="prometheus"
Apr 16 16:31:50.842531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842535    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="kube-rbac-proxy"
Apr 16 16:31:50.842995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842541    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="kube-rbac-proxy"
Apr 16 16:31:50.842995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842568    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde4edfa-4c1f-4006-8f70-12a437387527" containerName="console"
Apr 16 16:31:50.842995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842576    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde4edfa-4c1f-4006-8f70-12a437387527" containerName="console"
Apr 16 16:31:50.842995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842590    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="kube-rbac-proxy-web"
Apr 16 16:31:50.842995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842596    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="kube-rbac-proxy-web"
Apr 16 16:31:50.842995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842642    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="prometheus"
Apr 16 16:31:50.842995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842652    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="kube-rbac-proxy-thanos"
Apr 16 16:31:50.842995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842662    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="kube-rbac-proxy"
Apr 16 16:31:50.842995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842672    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="cde4edfa-4c1f-4006-8f70-12a437387527" containerName="console"
Apr 16 16:31:50.842995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842678    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="thanos-sidecar"
Apr 16 16:31:50.842995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842673    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3"} err="failed to get container status \"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3\": rpc error: code = NotFound desc = could not find container \"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3\": container with ID starting with 1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3 not found: ID does not exist"
Apr 16 16:31:50.842995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842703    2578 scope.go:117] "RemoveContainer" containerID="e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44"
Apr 16 16:31:50.842995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842686    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="config-reloader"
Apr 16 16:31:50.842995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842767    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" containerName="kube-rbac-proxy-web"
Apr 16 16:31:50.842995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842953    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44"} err="failed to get container status \"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44\": rpc error: code = NotFound desc = could not find container \"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44\": container with ID starting with e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44 not found: ID does not exist"
Apr 16 16:31:50.842995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.842969    2578 scope.go:117] "RemoveContainer" containerID="9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af"
Apr 16 16:31:50.843738 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.843184    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af"} err="failed to get container status \"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af\": rpc error: code = NotFound desc = could not find container \"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af\": container with ID starting with 9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af not found: ID does not exist"
Apr 16 16:31:50.843738 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.843197    2578 scope.go:117] "RemoveContainer" containerID="07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5"
Apr 16 16:31:50.843738 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.843403    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5"} err="failed to get container status \"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5\": rpc error: code = NotFound desc = could not find container \"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5\": container with ID starting with 07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5 not found: ID does not exist"
Apr 16 16:31:50.843738 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.843424    2578 scope.go:117] "RemoveContainer" containerID="68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1"
Apr 16 16:31:50.843738 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.843673    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1"} err="failed to get container status \"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1\": rpc error: code = NotFound desc = could not find container \"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1\": container with ID starting with 68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1 not found: ID does not exist"
Apr 16 16:31:50.843738 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.843694    2578 scope.go:117] "RemoveContainer" containerID="40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10"
Apr 16 16:31:50.844004 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.843870    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10"} err="failed to get container status \"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10\": rpc error: code = NotFound desc = could not find container \"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10\": container with ID starting with 40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10 not found: ID does not exist"
Apr 16 16:31:50.844004 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.843889    2578 scope.go:117] "RemoveContainer" containerID="d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7"
Apr 16 16:31:50.844106 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.844068    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7"} err="failed to get container status \"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7\": rpc error: code = NotFound desc = could not find container \"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7\": container with ID starting with d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7 not found: ID does not exist"
Apr 16 16:31:50.844106 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.844091    2578 scope.go:117] "RemoveContainer" containerID="1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3"
Apr 16 16:31:50.844477 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.844448    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3"} err="failed to get container status \"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3\": rpc error: code = NotFound desc = could not find container \"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3\": container with ID starting with 1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3 not found: ID does not exist"
Apr 16 16:31:50.844583 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.844479    2578 scope.go:117] "RemoveContainer" containerID="e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44"
Apr 16 16:31:50.844763 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.844741    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44"} err="failed to get container status \"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44\": rpc error: code = NotFound desc = could not find container \"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44\": container with ID starting with e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44 not found: ID does not exist"
Apr 16 16:31:50.844763 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.844762    2578 scope.go:117] "RemoveContainer" containerID="9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af"
Apr 16 16:31:50.846858 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.844990    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af"} err="failed to get container status \"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af\": rpc error: code = NotFound desc = could not find container \"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af\": container with ID starting with 9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af not found: ID does not exist"
Apr 16 16:31:50.846858 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.845012    2578 scope.go:117] "RemoveContainer" containerID="07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5"
Apr 16 16:31:50.846928 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.846859    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5"} err="failed to get container status \"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5\": rpc error: code = NotFound desc = could not find container \"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5\": container with ID starting with 07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5 not found: ID does not exist"
Apr 16 16:31:50.846928 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.846876    2578 scope.go:117] "RemoveContainer" containerID="68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1"
Apr 16 16:31:50.847156 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.847131    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1"} err="failed to get container status \"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1\": rpc error: code = NotFound desc = could not find container \"68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1\": container with ID starting with 68cd97f30d674c177d32fae6875bbb037427a66679b659332ac415489fbce2c1 not found: ID does not exist"
Apr 16 16:31:50.847156 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.847148    2578 scope.go:117] "RemoveContainer" containerID="40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10"
Apr 16 16:31:50.847438 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.847418    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10"} err="failed to get container status \"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10\": rpc error: code = NotFound desc = could not find container \"40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10\": container with ID starting with 40def66305bc3440ee33643cb4ed93664f1c8ea58da77be593266eed79002d10 not found: ID does not exist"
Apr 16 16:31:50.847518 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.847440    2578 scope.go:117] "RemoveContainer" containerID="d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7"
Apr 16 16:31:50.847706 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.847689    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7"} err="failed to get container status \"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7\": rpc error: code = NotFound desc = could not find container \"d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7\": container with ID starting with d78c4186b9e3b067b73f76e15a963ef98094a140bd3f5625e6ce93df3f06d8a7 not found: ID does not exist"
Apr 16 16:31:50.847769 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.847707    2578 scope.go:117] "RemoveContainer" containerID="1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3"
Apr 16 16:31:50.847921 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.847903    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3"} err="failed to get container status \"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3\": rpc error: code = NotFound desc = could not find container \"1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3\": container with ID starting with 1dbfe27cd1f577a174460d0258f13e73d4c1145b54eb074c7a9ed482a0bcf8d3 not found: ID does not exist"
Apr 16 16:31:50.847976 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.847923    2578 scope.go:117] "RemoveContainer" containerID="e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44"
Apr 16 16:31:50.848113 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.848095    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44"} err="failed to get container status \"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44\": rpc error: code = NotFound desc = could not find container \"e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44\": container with ID starting with e7aa8b482adf02ce85e2351ec8f60b8f91f7a182c851d2b16fe90ac3ee319f44 not found: ID does not exist"
Apr 16 16:31:50.848201 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.848114    2578 scope.go:117] "RemoveContainer" containerID="9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af"
Apr 16 16:31:50.848338 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.848319    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af"} err="failed to get container status \"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af\": rpc error: code = NotFound desc = could not find container \"9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af\": container with ID starting with 9372a7c8d68a03880c3e42f887304135acf78069a9248f9caa8966fcb07d74af not found: ID does not exist"
Apr 16 16:31:50.848384 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.848339    2578 scope.go:117] "RemoveContainer" containerID="07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5"
Apr 16 16:31:50.848525 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.848510    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5"} err="failed to get container status \"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5\": rpc error: code = NotFound desc = could not find container \"07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5\": container with ID starting with 07b091e25791006d60cf1147573445ab0705697c4dfc0515871c932af5d0c8f5 not found: ID does not exist"
Apr 16 16:31:50.849499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.849486    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.851747 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.851727    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Apr 16 16:31:50.851860 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.851844    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Apr 16 16:31:50.851915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.851891    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Apr 16 16:31:50.852254 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.852233    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Apr 16 16:31:50.852498 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.852268    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Apr 16 16:31:50.852498 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.852272    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Apr 16 16:31:50.852498 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.852315    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-auu6g6r71put3\""
Apr 16 16:31:50.852498 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.852280    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Apr 16 16:31:50.852498 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.852388    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Apr 16 16:31:50.852773 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.852748    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Apr 16 16:31:50.852876 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.852778    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Apr 16 16:31:50.852876 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.852842    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hcgqb\""
Apr 16 16:31:50.853046 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.853028    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Apr 16 16:31:50.855474 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.855436    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Apr 16 16:31:50.857192 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.857156    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 16 16:31:50.858824 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.858806    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Apr 16 16:31:50.936825 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.936801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.936825 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.936826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.936961 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.936845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.936961 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.936861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.936961 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.936881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.937064 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.936957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.937064 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.936992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.937064 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.937011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.937064 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.937059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.937179 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.937079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.937179 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.937097    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.937179 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.937118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.937179 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.937168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmm82\" (UniqueName: \"kubernetes.io/projected/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-kube-api-access-mmm82\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.937289 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.937190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.937289 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.937220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.937289 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.937248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-config\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.937289 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.937263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:50.937289 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:50.937279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.037960 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.037927    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-config\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.037960 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.037967    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.038084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.037992    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.038084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.038019    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.038084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.038040    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.038084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.038065    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.038258 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.038091    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.038258 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.038116    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.038258 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.038157    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.038258 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.038184    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.038258 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.038205    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.038258 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.038229    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.038531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.038260    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.038531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.038286    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.038531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.038318    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.038531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.038359    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mmm82\" (UniqueName: \"kubernetes.io/projected/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-kube-api-access-mmm82\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.038531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.038389    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.038531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.038427    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.039245 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.038906    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.041211 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.041019    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.041211 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.041039    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-config\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.041211 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.041109    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.041211 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.041106    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.041468 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.041226    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.041468 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.041315    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.041468 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.041378    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.041724 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.041702    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.042072 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.042019    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.042296 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.042272    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.042369 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.042308    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.042463 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.042441    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.043217 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.043193    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.043512 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.043495    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.043578 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.043521    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.044280 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.044261    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.048302 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.048285    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmm82\" (UniqueName: \"kubernetes.io/projected/ed3d8c6d-3bd0-4cc6-beb9-c5e737944966-kube-api-access-mmm82\") pod \"prometheus-k8s-0\" (UID: \"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.163363 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.163341    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hcgqb\""
Apr 16 16:31:51.172047 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.172031    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:31:51.236658 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.236626    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13a73e5c-19b8-43cd-9ad0-3db3153f7816" path="/var/lib/kubelet/pods/13a73e5c-19b8-43cd-9ad0-3db3153f7816/volumes"
Apr 16 16:31:51.302131 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.302057    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 16 16:31:51.304729 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:31:51.304703    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d8c6d_3bd0_4cc6_beb9_c5e737944966.slice/crio-042a080123e8cad6233f38fda987c0c21f5a862979d9c4b06d31c373618fcd29 WatchSource:0}: Error finding container 042a080123e8cad6233f38fda987c0c21f5a862979d9c4b06d31c373618fcd29: Status 404 returned error can't find the container with id 042a080123e8cad6233f38fda987c0c21f5a862979d9c4b06d31c373618fcd29
Apr 16 16:31:51.796686 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.796657    2578 generic.go:358] "Generic (PLEG): container finished" podID="ed3d8c6d-3bd0-4cc6-beb9-c5e737944966" containerID="fdba07844875c45364e87ef3aee7fbaea2391a7d72befffd99d11fd09818ebec" exitCode=0
Apr 16 16:31:51.796787 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.796733    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966","Type":"ContainerDied","Data":"fdba07844875c45364e87ef3aee7fbaea2391a7d72befffd99d11fd09818ebec"}
Apr 16 16:31:51.796787 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:51.796754    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966","Type":"ContainerStarted","Data":"042a080123e8cad6233f38fda987c0c21f5a862979d9c4b06d31c373618fcd29"}
Apr 16 16:31:52.799943 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.799915    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-787745798d-w7jnd"]
Apr 16 16:31:52.803322 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.803303    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.803429 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.803314    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966","Type":"ContainerStarted","Data":"67064511d2df010b2bf6269a1138dc85c9719763aba0346e1a70b427b90bc2e7"}
Apr 16 16:31:52.803429 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.803343    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966","Type":"ContainerStarted","Data":"78d3bfbe5322f799b8c7d8ee875cdf743390d85b7d4257e7ede6d3c6fca211c7"}
Apr 16 16:31:52.803429 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.803355    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966","Type":"ContainerStarted","Data":"e9f732d929683d4cc05fcdec4e3b91b812fe83fc07148630fd816d69534473ee"}
Apr 16 16:31:52.803429 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.803364    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966","Type":"ContainerStarted","Data":"9ee4f6d28153a25ca6d20b411f1cc1d7c9eb0df93e01a30b28855b7019d6c96f"}
Apr 16 16:31:52.803429 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.803375    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966","Type":"ContainerStarted","Data":"f31c5e4ee1f181429a74b12a3299f24024fa9f1cd33ef42cfe933007379c4cd7"}
Apr 16 16:31:52.803429 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.803387    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed3d8c6d-3bd0-4cc6-beb9-c5e737944966","Type":"ContainerStarted","Data":"30c98297d4e92399bb030f1b739fe0a65fbb9ae1c2d66a9df4c9a77b6065e44e"}
Apr 16 16:31:52.813318 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.813296    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-787745798d-w7jnd"]
Apr 16 16:31:52.851333 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.851307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-oauth-config\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.851333 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.851335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-service-ca\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.851540 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.851455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwfsw\" (UniqueName: \"kubernetes.io/projected/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-kube-api-access-qwfsw\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.851813 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.851635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-trusted-ca-bundle\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.851813 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.851686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-serving-cert\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.851813 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.851770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-oauth-serving-cert\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.851972 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.851827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-config\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.860756 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.860719    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.860708172 podStartE2EDuration="2.860708172s" podCreationTimestamp="2026-04-16 16:31:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:31:52.858835169 +0000 UTC m=+122.256051569" watchObservedRunningTime="2026-04-16 16:31:52.860708172 +0000 UTC m=+122.257924550"
Apr 16 16:31:52.953017 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.952981    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-oauth-serving-cert\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.953159 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.953033    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-config\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.953159 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.953078    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-oauth-config\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.953159 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.953101    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-service-ca\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.953159 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.953148    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qwfsw\" (UniqueName: \"kubernetes.io/projected/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-kube-api-access-qwfsw\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.953365 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.953170    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-trusted-ca-bundle\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.953365 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.953197    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-serving-cert\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.954143 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.954118    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-oauth-serving-cert\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.954304 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.954282    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-trusted-ca-bundle\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.954658 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.954634    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-config\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.954758 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.954738    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-service-ca\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.955869 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.955850    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-oauth-config\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.956076 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.956054    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-serving-cert\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:52.960193 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:52.960170    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwfsw\" (UniqueName: \"kubernetes.io/projected/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-kube-api-access-qwfsw\") pod \"console-787745798d-w7jnd\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") " pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:53.111696 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:53.111617    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:31:53.241699 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:53.241666    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-787745798d-w7jnd"]
Apr 16 16:31:53.245182 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:31:53.245145    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9b7a67_1c41_45b1_90b7_e1762cbc887b.slice/crio-1ad91d56578040dbafb24f67c68fa241f248cd6d6b4fe116b1e44b2832fabc0d WatchSource:0}: Error finding container 1ad91d56578040dbafb24f67c68fa241f248cd6d6b4fe116b1e44b2832fabc0d: Status 404 returned error can't find the container with id 1ad91d56578040dbafb24f67c68fa241f248cd6d6b4fe116b1e44b2832fabc0d
Apr 16 16:31:53.807453 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:53.807419    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-787745798d-w7jnd" event={"ID":"0c9b7a67-1c41-45b1-90b7-e1762cbc887b","Type":"ContainerStarted","Data":"82a0a6199e8a8a86820b0438336e8793f1f8817ad249be68b50ac11c5b9f44d4"}
Apr 16 16:31:53.807453 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:53.807455    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-787745798d-w7jnd" event={"ID":"0c9b7a67-1c41-45b1-90b7-e1762cbc887b","Type":"ContainerStarted","Data":"1ad91d56578040dbafb24f67c68fa241f248cd6d6b4fe116b1e44b2832fabc0d"}
Apr 16 16:31:53.822788 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:53.822730    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-787745798d-w7jnd" podStartSLOduration=1.822716008 podStartE2EDuration="1.822716008s" podCreationTimestamp="2026-04-16 16:31:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:31:53.822470037 +0000 UTC m=+123.219686428" watchObservedRunningTime="2026-04-16 16:31:53.822716008 +0000 UTC m=+123.219932395"
Apr 16 16:31:56.172198 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:31:56.172173    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:32:03.112626 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:03.112599    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:32:03.113036 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:03.112840    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:32:03.117276 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:03.117257    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:32:03.839369 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:03.839336    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:32:03.882256 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:03.882220    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-644dcd4585-bnnh8"]
Apr 16 16:32:28.901074 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:28.901022    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-644dcd4585-bnnh8" podUID="38ce9b1e-ae3d-48f8-85b2-9953825acddf" containerName="console" containerID="cri-o://0224c0c517187f5ee1f928247d2d44252f81c0686a1364a345f5ff7c74100424" gracePeriod=15
Apr 16 16:32:29.130059 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.130036    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-644dcd4585-bnnh8_38ce9b1e-ae3d-48f8-85b2-9953825acddf/console/0.log"
Apr 16 16:32:29.130181 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.130109    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:32:29.210833 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.210805    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-config\") pod \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") "
Apr 16 16:32:29.210967 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.210838    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-oauth-config\") pod \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") "
Apr 16 16:32:29.210967 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.210869    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-serving-cert\") pod \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") "
Apr 16 16:32:29.210967 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.210910    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-oauth-serving-cert\") pod \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") "
Apr 16 16:32:29.210967 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.210930    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-trusted-ca-bundle\") pod \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") "
Apr 16 16:32:29.211182 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.210979    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwc2s\" (UniqueName: \"kubernetes.io/projected/38ce9b1e-ae3d-48f8-85b2-9953825acddf-kube-api-access-dwc2s\") pod \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") "
Apr 16 16:32:29.211182 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.211023    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-service-ca\") pod \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\" (UID: \"38ce9b1e-ae3d-48f8-85b2-9953825acddf\") "
Apr 16 16:32:29.211328 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.211306    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-config" (OuterVolumeSpecName: "console-config") pod "38ce9b1e-ae3d-48f8-85b2-9953825acddf" (UID: "38ce9b1e-ae3d-48f8-85b2-9953825acddf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:32:29.211444 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.211414    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "38ce9b1e-ae3d-48f8-85b2-9953825acddf" (UID: "38ce9b1e-ae3d-48f8-85b2-9953825acddf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:32:29.211509 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.211366    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "38ce9b1e-ae3d-48f8-85b2-9953825acddf" (UID: "38ce9b1e-ae3d-48f8-85b2-9953825acddf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:32:29.211580 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.211503    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-service-ca" (OuterVolumeSpecName: "service-ca") pod "38ce9b1e-ae3d-48f8-85b2-9953825acddf" (UID: "38ce9b1e-ae3d-48f8-85b2-9953825acddf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:32:29.212952 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.212928    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "38ce9b1e-ae3d-48f8-85b2-9953825acddf" (UID: "38ce9b1e-ae3d-48f8-85b2-9953825acddf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:32:29.213046 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.213023    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "38ce9b1e-ae3d-48f8-85b2-9953825acddf" (UID: "38ce9b1e-ae3d-48f8-85b2-9953825acddf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:32:29.213116 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.213097    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ce9b1e-ae3d-48f8-85b2-9953825acddf-kube-api-access-dwc2s" (OuterVolumeSpecName: "kube-api-access-dwc2s") pod "38ce9b1e-ae3d-48f8-85b2-9953825acddf" (UID: "38ce9b1e-ae3d-48f8-85b2-9953825acddf"). InnerVolumeSpecName "kube-api-access-dwc2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:32:29.312013 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.311967    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-config\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:32:29.312013 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.311986    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-oauth-config\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:32:29.312013 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.311996    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38ce9b1e-ae3d-48f8-85b2-9953825acddf-console-serving-cert\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:32:29.312013 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.312007    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-oauth-serving-cert\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:32:29.312193 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.312017    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-trusted-ca-bundle\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:32:29.312193 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.312026    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwc2s\" (UniqueName: \"kubernetes.io/projected/38ce9b1e-ae3d-48f8-85b2-9953825acddf-kube-api-access-dwc2s\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:32:29.312193 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.312034    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38ce9b1e-ae3d-48f8-85b2-9953825acddf-service-ca\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:32:29.906223 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.906197    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-644dcd4585-bnnh8_38ce9b1e-ae3d-48f8-85b2-9953825acddf/console/0.log"
Apr 16 16:32:29.906650 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.906237    2578 generic.go:358] "Generic (PLEG): container finished" podID="38ce9b1e-ae3d-48f8-85b2-9953825acddf" containerID="0224c0c517187f5ee1f928247d2d44252f81c0686a1364a345f5ff7c74100424" exitCode=2
Apr 16 16:32:29.906650 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.906282    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644dcd4585-bnnh8" event={"ID":"38ce9b1e-ae3d-48f8-85b2-9953825acddf","Type":"ContainerDied","Data":"0224c0c517187f5ee1f928247d2d44252f81c0686a1364a345f5ff7c74100424"}
Apr 16 16:32:29.906650 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.906300    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644dcd4585-bnnh8"
Apr 16 16:32:29.906650 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.906308    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644dcd4585-bnnh8" event={"ID":"38ce9b1e-ae3d-48f8-85b2-9953825acddf","Type":"ContainerDied","Data":"31eddd48c38d4a493c365aa48fdbee2d03ccf620abd00f0550a6c2a40544a0e6"}
Apr 16 16:32:29.906650 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.906328    2578 scope.go:117] "RemoveContainer" containerID="0224c0c517187f5ee1f928247d2d44252f81c0686a1364a345f5ff7c74100424"
Apr 16 16:32:29.913777 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.913760    2578 scope.go:117] "RemoveContainer" containerID="0224c0c517187f5ee1f928247d2d44252f81c0686a1364a345f5ff7c74100424"
Apr 16 16:32:29.914060 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:32:29.914015    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0224c0c517187f5ee1f928247d2d44252f81c0686a1364a345f5ff7c74100424\": container with ID starting with 0224c0c517187f5ee1f928247d2d44252f81c0686a1364a345f5ff7c74100424 not found: ID does not exist" containerID="0224c0c517187f5ee1f928247d2d44252f81c0686a1364a345f5ff7c74100424"
Apr 16 16:32:29.914060 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.914039    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0224c0c517187f5ee1f928247d2d44252f81c0686a1364a345f5ff7c74100424"} err="failed to get container status \"0224c0c517187f5ee1f928247d2d44252f81c0686a1364a345f5ff7c74100424\": rpc error: code = NotFound desc = could not find container \"0224c0c517187f5ee1f928247d2d44252f81c0686a1364a345f5ff7c74100424\": container with ID starting with 0224c0c517187f5ee1f928247d2d44252f81c0686a1364a345f5ff7c74100424 not found: ID does not exist"
Apr 16 16:32:29.921859 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.921835    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-644dcd4585-bnnh8"]
Apr 16 16:32:29.925743 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:29.925726    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-644dcd4585-bnnh8"]
Apr 16 16:32:31.234034 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:31.234004    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38ce9b1e-ae3d-48f8-85b2-9953825acddf" path="/var/lib/kubelet/pods/38ce9b1e-ae3d-48f8-85b2-9953825acddf/volumes"
Apr 16 16:32:51.172843 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:51.172813    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:32:51.187813 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:51.187790    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:32:51.978345 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:32:51.978319    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Apr 16 16:33:25.157780 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.157754    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-64cc66665-ndb47"]
Apr 16 16:33:25.158256 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.158041    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ce9b1e-ae3d-48f8-85b2-9953825acddf" containerName="console"
Apr 16 16:33:25.158256 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.158053    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ce9b1e-ae3d-48f8-85b2-9953825acddf" containerName="console"
Apr 16 16:33:25.158256 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.158110    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="38ce9b1e-ae3d-48f8-85b2-9953825acddf" containerName="console"
Apr 16 16:33:25.160995 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.160978    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.171419 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.171396    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64cc66665-ndb47"]
Apr 16 16:33:25.190749 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.190729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-trusted-ca-bundle\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.190852 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.190756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-oauth-config\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.190852 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.190788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-config\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.190852 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.190845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-serving-cert\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.190978 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.190874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-service-ca\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.190978 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.190898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-oauth-serving-cert\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.190978 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.190915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgr4z\" (UniqueName: \"kubernetes.io/projected/bad3ae4b-579c-4f82-a6b2-16416b11f28e-kube-api-access-pgr4z\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.291952 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.291929    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-config\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.292060 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.291961    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-serving-cert\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.292060 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.291980    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-service-ca\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.292060 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.292009    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-oauth-serving-cert\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.292060 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.292034    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pgr4z\" (UniqueName: \"kubernetes.io/projected/bad3ae4b-579c-4f82-a6b2-16416b11f28e-kube-api-access-pgr4z\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.292265 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.292072    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-trusted-ca-bundle\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.292265 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.292097    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-oauth-config\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.292662 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.292637    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-config\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.292769 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.292713    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-oauth-serving-cert\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.292891 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.292874    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-service-ca\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.292956 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.292901    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-trusted-ca-bundle\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.294467 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.294447    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-oauth-config\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.294570 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.294521    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-serving-cert\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.303644 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.303625    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgr4z\" (UniqueName: \"kubernetes.io/projected/bad3ae4b-579c-4f82-a6b2-16416b11f28e-kube-api-access-pgr4z\") pod \"console-64cc66665-ndb47\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") " pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.451057 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.451034    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-bk4tn"]
Apr 16 16:33:25.455424 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.455410    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bk4tn"
Apr 16 16:33:25.458058 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.458038    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Apr 16 16:33:25.463284 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.463262    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bk4tn"]
Apr 16 16:33:25.469348 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.469326    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:25.492846 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.492825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1bb21574-8866-4f9f-a9e3-9fb3e776094e-kubelet-config\") pod \"global-pull-secret-syncer-bk4tn\" (UID: \"1bb21574-8866-4f9f-a9e3-9fb3e776094e\") " pod="kube-system/global-pull-secret-syncer-bk4tn"
Apr 16 16:33:25.492962 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.492870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bb21574-8866-4f9f-a9e3-9fb3e776094e-original-pull-secret\") pod \"global-pull-secret-syncer-bk4tn\" (UID: \"1bb21574-8866-4f9f-a9e3-9fb3e776094e\") " pod="kube-system/global-pull-secret-syncer-bk4tn"
Apr 16 16:33:25.492962 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.492934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1bb21574-8866-4f9f-a9e3-9fb3e776094e-dbus\") pod \"global-pull-secret-syncer-bk4tn\" (UID: \"1bb21574-8866-4f9f-a9e3-9fb3e776094e\") " pod="kube-system/global-pull-secret-syncer-bk4tn"
Apr 16 16:33:25.583655 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.583627    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64cc66665-ndb47"]
Apr 16 16:33:25.586268 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:33:25.586240    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad3ae4b_579c_4f82_a6b2_16416b11f28e.slice/crio-f18bbdc286cd878c960eee972bd52e897e1a6a98c0fb90917df885f8a6c0d65f WatchSource:0}: Error finding container f18bbdc286cd878c960eee972bd52e897e1a6a98c0fb90917df885f8a6c0d65f: Status 404 returned error can't find the container with id f18bbdc286cd878c960eee972bd52e897e1a6a98c0fb90917df885f8a6c0d65f
Apr 16 16:33:25.594117 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.594097    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1bb21574-8866-4f9f-a9e3-9fb3e776094e-kubelet-config\") pod \"global-pull-secret-syncer-bk4tn\" (UID: \"1bb21574-8866-4f9f-a9e3-9fb3e776094e\") " pod="kube-system/global-pull-secret-syncer-bk4tn"
Apr 16 16:33:25.594194 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.594142    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bb21574-8866-4f9f-a9e3-9fb3e776094e-original-pull-secret\") pod \"global-pull-secret-syncer-bk4tn\" (UID: \"1bb21574-8866-4f9f-a9e3-9fb3e776094e\") " pod="kube-system/global-pull-secret-syncer-bk4tn"
Apr 16 16:33:25.594194 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.594181    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1bb21574-8866-4f9f-a9e3-9fb3e776094e-dbus\") pod \"global-pull-secret-syncer-bk4tn\" (UID: \"1bb21574-8866-4f9f-a9e3-9fb3e776094e\") " pod="kube-system/global-pull-secret-syncer-bk4tn"
Apr 16 16:33:25.594285 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.594232    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1bb21574-8866-4f9f-a9e3-9fb3e776094e-kubelet-config\") pod \"global-pull-secret-syncer-bk4tn\" (UID: \"1bb21574-8866-4f9f-a9e3-9fb3e776094e\") " pod="kube-system/global-pull-secret-syncer-bk4tn"
Apr 16 16:33:25.594383 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.594366    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1bb21574-8866-4f9f-a9e3-9fb3e776094e-dbus\") pod \"global-pull-secret-syncer-bk4tn\" (UID: \"1bb21574-8866-4f9f-a9e3-9fb3e776094e\") " pod="kube-system/global-pull-secret-syncer-bk4tn"
Apr 16 16:33:25.596249 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.596222    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bb21574-8866-4f9f-a9e3-9fb3e776094e-original-pull-secret\") pod \"global-pull-secret-syncer-bk4tn\" (UID: \"1bb21574-8866-4f9f-a9e3-9fb3e776094e\") " pod="kube-system/global-pull-secret-syncer-bk4tn"
Apr 16 16:33:25.764790 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.764720    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bk4tn"
Apr 16 16:33:25.872163 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:25.872128    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bk4tn"]
Apr 16 16:33:25.874991 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:33:25.874964    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb21574_8866_4f9f_a9e3_9fb3e776094e.slice/crio-ea1c125656bb4d1634642a584b8cdf10fdce84c3c47436d78b36dd6564a470a3 WatchSource:0}: Error finding container ea1c125656bb4d1634642a584b8cdf10fdce84c3c47436d78b36dd6564a470a3: Status 404 returned error can't find the container with id ea1c125656bb4d1634642a584b8cdf10fdce84c3c47436d78b36dd6564a470a3
Apr 16 16:33:26.051238 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:26.051168    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cc66665-ndb47" event={"ID":"bad3ae4b-579c-4f82-a6b2-16416b11f28e","Type":"ContainerStarted","Data":"e2296c9aae1209ddcf2e444dce9e589a68f06156328b5e54201a0c2ec6b63553"}
Apr 16 16:33:26.051238 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:26.051220    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cc66665-ndb47" event={"ID":"bad3ae4b-579c-4f82-a6b2-16416b11f28e","Type":"ContainerStarted","Data":"f18bbdc286cd878c960eee972bd52e897e1a6a98c0fb90917df885f8a6c0d65f"}
Apr 16 16:33:26.052130 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:26.052111    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bk4tn" event={"ID":"1bb21574-8866-4f9f-a9e3-9fb3e776094e","Type":"ContainerStarted","Data":"ea1c125656bb4d1634642a584b8cdf10fdce84c3c47436d78b36dd6564a470a3"}
Apr 16 16:33:26.067983 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:26.067944    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-64cc66665-ndb47" podStartSLOduration=1.067932012 podStartE2EDuration="1.067932012s" podCreationTimestamp="2026-04-16 16:33:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:33:26.066897949 +0000 UTC m=+215.464114336" watchObservedRunningTime="2026-04-16 16:33:26.067932012 +0000 UTC m=+215.465148400"
Apr 16 16:33:30.065646 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:30.065611    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bk4tn" event={"ID":"1bb21574-8866-4f9f-a9e3-9fb3e776094e","Type":"ContainerStarted","Data":"b7438f6536e49aff15d8119980e39fc86ff866fa3bdc1cb903139590296b3181"}
Apr 16 16:33:30.078507 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:30.078459    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bk4tn" podStartSLOduration=1.281272197 podStartE2EDuration="5.078442054s" podCreationTimestamp="2026-04-16 16:33:25 +0000 UTC" firstStartedPulling="2026-04-16 16:33:25.876588198 +0000 UTC m=+215.273804564" lastFinishedPulling="2026-04-16 16:33:29.673758055 +0000 UTC m=+219.070974421" observedRunningTime="2026-04-16 16:33:30.078140368 +0000 UTC m=+219.475356780" watchObservedRunningTime="2026-04-16 16:33:30.078442054 +0000 UTC m=+219.475658442"
Apr 16 16:33:35.470498 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:35.470467    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:35.470498 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:35.470498    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:35.475141 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:35.475118    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:36.086388 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:36.086356    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:33:36.126633 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:33:36.126605    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-787745798d-w7jnd"]
Apr 16 16:34:01.145745 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.145644    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-787745798d-w7jnd" podUID="0c9b7a67-1c41-45b1-90b7-e1762cbc887b" containerName="console" containerID="cri-o://82a0a6199e8a8a86820b0438336e8793f1f8817ad249be68b50ac11c5b9f44d4" gracePeriod=15
Apr 16 16:34:01.374154 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.374135    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-787745798d-w7jnd_0c9b7a67-1c41-45b1-90b7-e1762cbc887b/console/0.log"
Apr 16 16:34:01.374251 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.374194    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:34:01.446179 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.446121    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-trusted-ca-bundle\") pod \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") "
Apr 16 16:34:01.446179 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.446174    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-service-ca\") pod \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") "
Apr 16 16:34:01.446349 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.446216    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwfsw\" (UniqueName: \"kubernetes.io/projected/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-kube-api-access-qwfsw\") pod \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") "
Apr 16 16:34:01.446349 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.446244    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-config\") pod \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") "
Apr 16 16:34:01.446349 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.446303    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-oauth-config\") pod \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") "
Apr 16 16:34:01.446349 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.446329    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-serving-cert\") pod \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") "
Apr 16 16:34:01.446534 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.446352    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-oauth-serving-cert\") pod \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\" (UID: \"0c9b7a67-1c41-45b1-90b7-e1762cbc887b\") "
Apr 16 16:34:01.446534 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.446505    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0c9b7a67-1c41-45b1-90b7-e1762cbc887b" (UID: "0c9b7a67-1c41-45b1-90b7-e1762cbc887b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:34:01.446667 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.446631    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-trusted-ca-bundle\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:34:01.446667 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.446640    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-service-ca" (OuterVolumeSpecName: "service-ca") pod "0c9b7a67-1c41-45b1-90b7-e1762cbc887b" (UID: "0c9b7a67-1c41-45b1-90b7-e1762cbc887b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:34:01.446843 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.446817    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-config" (OuterVolumeSpecName: "console-config") pod "0c9b7a67-1c41-45b1-90b7-e1762cbc887b" (UID: "0c9b7a67-1c41-45b1-90b7-e1762cbc887b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:34:01.446904 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.446845    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0c9b7a67-1c41-45b1-90b7-e1762cbc887b" (UID: "0c9b7a67-1c41-45b1-90b7-e1762cbc887b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:34:01.448316 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.448292    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0c9b7a67-1c41-45b1-90b7-e1762cbc887b" (UID: "0c9b7a67-1c41-45b1-90b7-e1762cbc887b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:34:01.448412 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.448391    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0c9b7a67-1c41-45b1-90b7-e1762cbc887b" (UID: "0c9b7a67-1c41-45b1-90b7-e1762cbc887b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:34:01.448449 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.448423    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-kube-api-access-qwfsw" (OuterVolumeSpecName: "kube-api-access-qwfsw") pod "0c9b7a67-1c41-45b1-90b7-e1762cbc887b" (UID: "0c9b7a67-1c41-45b1-90b7-e1762cbc887b"). InnerVolumeSpecName "kube-api-access-qwfsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:34:01.547779 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.547757    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-service-ca\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:34:01.547779 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.547779    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwfsw\" (UniqueName: \"kubernetes.io/projected/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-kube-api-access-qwfsw\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:34:01.547947 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.547790    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-config\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:34:01.547947 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.547799    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-oauth-config\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:34:01.547947 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.547807    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-console-serving-cert\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:34:01.547947 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:01.547816    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c9b7a67-1c41-45b1-90b7-e1762cbc887b-oauth-serving-cert\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:34:02.157308 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:02.157282    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-787745798d-w7jnd_0c9b7a67-1c41-45b1-90b7-e1762cbc887b/console/0.log"
Apr 16 16:34:02.157720 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:02.157324    2578 generic.go:358] "Generic (PLEG): container finished" podID="0c9b7a67-1c41-45b1-90b7-e1762cbc887b" containerID="82a0a6199e8a8a86820b0438336e8793f1f8817ad249be68b50ac11c5b9f44d4" exitCode=2
Apr 16 16:34:02.157720 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:02.157357    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-787745798d-w7jnd" event={"ID":"0c9b7a67-1c41-45b1-90b7-e1762cbc887b","Type":"ContainerDied","Data":"82a0a6199e8a8a86820b0438336e8793f1f8817ad249be68b50ac11c5b9f44d4"}
Apr 16 16:34:02.157720 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:02.157397    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-787745798d-w7jnd"
Apr 16 16:34:02.157720 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:02.157410    2578 scope.go:117] "RemoveContainer" containerID="82a0a6199e8a8a86820b0438336e8793f1f8817ad249be68b50ac11c5b9f44d4"
Apr 16 16:34:02.157720 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:02.157400    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-787745798d-w7jnd" event={"ID":"0c9b7a67-1c41-45b1-90b7-e1762cbc887b","Type":"ContainerDied","Data":"1ad91d56578040dbafb24f67c68fa241f248cd6d6b4fe116b1e44b2832fabc0d"}
Apr 16 16:34:02.165616 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:02.165368    2578 scope.go:117] "RemoveContainer" containerID="82a0a6199e8a8a86820b0438336e8793f1f8817ad249be68b50ac11c5b9f44d4"
Apr 16 16:34:02.165712 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:34:02.165672    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a0a6199e8a8a86820b0438336e8793f1f8817ad249be68b50ac11c5b9f44d4\": container with ID starting with 82a0a6199e8a8a86820b0438336e8793f1f8817ad249be68b50ac11c5b9f44d4 not found: ID does not exist" containerID="82a0a6199e8a8a86820b0438336e8793f1f8817ad249be68b50ac11c5b9f44d4"
Apr 16 16:34:02.165774 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:02.165710    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a0a6199e8a8a86820b0438336e8793f1f8817ad249be68b50ac11c5b9f44d4"} err="failed to get container status \"82a0a6199e8a8a86820b0438336e8793f1f8817ad249be68b50ac11c5b9f44d4\": rpc error: code = NotFound desc = could not find container \"82a0a6199e8a8a86820b0438336e8793f1f8817ad249be68b50ac11c5b9f44d4\": container with ID starting with 82a0a6199e8a8a86820b0438336e8793f1f8817ad249be68b50ac11c5b9f44d4 not found: ID does not exist"
Apr 16 16:34:02.178366 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:02.178348    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-787745798d-w7jnd"]
Apr 16 16:34:02.181734 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:02.181715    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-787745798d-w7jnd"]
Apr 16 16:34:03.234918 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:03.234886    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c9b7a67-1c41-45b1-90b7-e1762cbc887b" path="/var/lib/kubelet/pods/0c9b7a67-1c41-45b1-90b7-e1762cbc887b/volumes"
Apr 16 16:34:51.124513 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:51.124480    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 16:34:51.125044 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:34:51.124978    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 16:36:31.829376 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:31.829344    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-l86r8"]
Apr 16 16:36:31.829785 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:31.829636    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c9b7a67-1c41-45b1-90b7-e1762cbc887b" containerName="console"
Apr 16 16:36:31.829785 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:31.829648    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c9b7a67-1c41-45b1-90b7-e1762cbc887b" containerName="console"
Apr 16 16:36:31.829785 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:31.829713    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="0c9b7a67-1c41-45b1-90b7-e1762cbc887b" containerName="console"
Apr 16 16:36:31.832451 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:31.832430    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-l86r8"
Apr 16 16:36:31.834717 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:31.834696    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Apr 16 16:36:31.834820 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:31.834731    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Apr 16 16:36:31.834869 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:31.834846    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-mhxsv\""
Apr 16 16:36:31.835742 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:31.835727    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Apr 16 16:36:31.843347 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:31.843322    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-l86r8"]
Apr 16 16:36:31.952251 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:31.952227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/097dd376-9dd2-4b76-8a0e-16f5ec05e180-tls-certs\") pod \"model-serving-api-86f7b4b499-l86r8\" (UID: \"097dd376-9dd2-4b76-8a0e-16f5ec05e180\") " pod="kserve/model-serving-api-86f7b4b499-l86r8"
Apr 16 16:36:31.952353 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:31.952274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2w2h\" (UniqueName: \"kubernetes.io/projected/097dd376-9dd2-4b76-8a0e-16f5ec05e180-kube-api-access-q2w2h\") pod \"model-serving-api-86f7b4b499-l86r8\" (UID: \"097dd376-9dd2-4b76-8a0e-16f5ec05e180\") " pod="kserve/model-serving-api-86f7b4b499-l86r8"
Apr 16 16:36:32.053394 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:32.053370    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/097dd376-9dd2-4b76-8a0e-16f5ec05e180-tls-certs\") pod \"model-serving-api-86f7b4b499-l86r8\" (UID: \"097dd376-9dd2-4b76-8a0e-16f5ec05e180\") " pod="kserve/model-serving-api-86f7b4b499-l86r8"
Apr 16 16:36:32.053513 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:32.053441    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q2w2h\" (UniqueName: \"kubernetes.io/projected/097dd376-9dd2-4b76-8a0e-16f5ec05e180-kube-api-access-q2w2h\") pod \"model-serving-api-86f7b4b499-l86r8\" (UID: \"097dd376-9dd2-4b76-8a0e-16f5ec05e180\") " pod="kserve/model-serving-api-86f7b4b499-l86r8"
Apr 16 16:36:32.053611 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:36:32.053521    2578 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Apr 16 16:36:32.053611 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:36:32.053599    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/097dd376-9dd2-4b76-8a0e-16f5ec05e180-tls-certs podName:097dd376-9dd2-4b76-8a0e-16f5ec05e180 nodeName:}" failed. No retries permitted until 2026-04-16 16:36:32.553580901 +0000 UTC m=+401.950797288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/097dd376-9dd2-4b76-8a0e-16f5ec05e180-tls-certs") pod "model-serving-api-86f7b4b499-l86r8" (UID: "097dd376-9dd2-4b76-8a0e-16f5ec05e180") : secret "model-serving-api-tls" not found
Apr 16 16:36:32.063289 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:32.063270    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2w2h\" (UniqueName: \"kubernetes.io/projected/097dd376-9dd2-4b76-8a0e-16f5ec05e180-kube-api-access-q2w2h\") pod \"model-serving-api-86f7b4b499-l86r8\" (UID: \"097dd376-9dd2-4b76-8a0e-16f5ec05e180\") " pod="kserve/model-serving-api-86f7b4b499-l86r8"
Apr 16 16:36:32.556631 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:32.556604    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/097dd376-9dd2-4b76-8a0e-16f5ec05e180-tls-certs\") pod \"model-serving-api-86f7b4b499-l86r8\" (UID: \"097dd376-9dd2-4b76-8a0e-16f5ec05e180\") " pod="kserve/model-serving-api-86f7b4b499-l86r8"
Apr 16 16:36:32.558934 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:32.558910    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/097dd376-9dd2-4b76-8a0e-16f5ec05e180-tls-certs\") pod \"model-serving-api-86f7b4b499-l86r8\" (UID: \"097dd376-9dd2-4b76-8a0e-16f5ec05e180\") " pod="kserve/model-serving-api-86f7b4b499-l86r8"
Apr 16 16:36:32.744215 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:32.744188    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-l86r8"
Apr 16 16:36:32.855690 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:32.855663    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-l86r8"]
Apr 16 16:36:32.858926 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:36:32.858893    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097dd376_9dd2_4b76_8a0e_16f5ec05e180.slice/crio-a48ef43af6acb9be7be6f1bdb46c0f9603797bbe488eca8c203480a0d74f4e62 WatchSource:0}: Error finding container a48ef43af6acb9be7be6f1bdb46c0f9603797bbe488eca8c203480a0d74f4e62: Status 404 returned error can't find the container with id a48ef43af6acb9be7be6f1bdb46c0f9603797bbe488eca8c203480a0d74f4e62
Apr 16 16:36:32.860478 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:32.860463    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 16:36:33.558047 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:33.558001    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-l86r8" event={"ID":"097dd376-9dd2-4b76-8a0e-16f5ec05e180","Type":"ContainerStarted","Data":"a48ef43af6acb9be7be6f1bdb46c0f9603797bbe488eca8c203480a0d74f4e62"}
Apr 16 16:36:35.565460 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:35.565432    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-l86r8" event={"ID":"097dd376-9dd2-4b76-8a0e-16f5ec05e180","Type":"ContainerStarted","Data":"4725ec32d0836dc266f1dd12964f018553c513af718273c483d3899a52244bac"}
Apr 16 16:36:35.565808 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:35.565587    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-l86r8"
Apr 16 16:36:35.580977 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:35.580930    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-l86r8" podStartSLOduration=2.279285238 podStartE2EDuration="4.580919743s" podCreationTimestamp="2026-04-16 16:36:31 +0000 UTC" firstStartedPulling="2026-04-16 16:36:32.860601464 +0000 UTC m=+402.257817831" lastFinishedPulling="2026-04-16 16:36:35.162235966 +0000 UTC m=+404.559452336" observedRunningTime="2026-04-16 16:36:35.579845387 +0000 UTC m=+404.977061778" watchObservedRunningTime="2026-04-16 16:36:35.580919743 +0000 UTC m=+404.978136135"
Apr 16 16:36:46.572879 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:46.572853    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-l86r8"
Apr 16 16:36:48.206174 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:48.206145    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-wxvfp"]
Apr 16 16:36:48.209348 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:48.209333    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-wxvfp"
Apr 16 16:36:48.211621 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:48.211600    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-x4ndd\""
Apr 16 16:36:48.211743 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:48.211601    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Apr 16 16:36:48.215069 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:48.215047    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-wxvfp"]
Apr 16 16:36:48.372109 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:48.372081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nsx8\" (UniqueName: \"kubernetes.io/projected/b8d03003-d679-4684-b987-75a4ec1a678f-kube-api-access-5nsx8\") pod \"s3-init-wxvfp\" (UID: \"b8d03003-d679-4684-b987-75a4ec1a678f\") " pod="kserve/s3-init-wxvfp"
Apr 16 16:36:48.472966 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:48.472906    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5nsx8\" (UniqueName: \"kubernetes.io/projected/b8d03003-d679-4684-b987-75a4ec1a678f-kube-api-access-5nsx8\") pod \"s3-init-wxvfp\" (UID: \"b8d03003-d679-4684-b987-75a4ec1a678f\") " pod="kserve/s3-init-wxvfp"
Apr 16 16:36:48.480880 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:48.480858    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nsx8\" (UniqueName: \"kubernetes.io/projected/b8d03003-d679-4684-b987-75a4ec1a678f-kube-api-access-5nsx8\") pod \"s3-init-wxvfp\" (UID: \"b8d03003-d679-4684-b987-75a4ec1a678f\") " pod="kserve/s3-init-wxvfp"
Apr 16 16:36:48.533506 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:48.533488    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-wxvfp"
Apr 16 16:36:48.647695 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:48.647672    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-wxvfp"]
Apr 16 16:36:48.650207 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:36:48.650182    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8d03003_d679_4684_b987_75a4ec1a678f.slice/crio-fd0a4bbea34479fcafadbf02ceb0244cf2501d39e98325c0786db05feaf1ee78 WatchSource:0}: Error finding container fd0a4bbea34479fcafadbf02ceb0244cf2501d39e98325c0786db05feaf1ee78: Status 404 returned error can't find the container with id fd0a4bbea34479fcafadbf02ceb0244cf2501d39e98325c0786db05feaf1ee78
Apr 16 16:36:49.602499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:49.602454    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-wxvfp" event={"ID":"b8d03003-d679-4684-b987-75a4ec1a678f","Type":"ContainerStarted","Data":"fd0a4bbea34479fcafadbf02ceb0244cf2501d39e98325c0786db05feaf1ee78"}
Apr 16 16:36:52.988484 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:52.988459    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Apr 16 16:36:53.614827 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:53.614795    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-wxvfp" event={"ID":"b8d03003-d679-4684-b987-75a4ec1a678f","Type":"ContainerStarted","Data":"8139e77a4377315359b62146d40ad79dc51be81ee078e8de2165c98e8c533ea5"}
Apr 16 16:36:53.628112 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:53.628036    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-wxvfp" podStartSLOduration=1.294215439 podStartE2EDuration="5.628024748s" podCreationTimestamp="2026-04-16 16:36:48 +0000 UTC" firstStartedPulling="2026-04-16 16:36:48.652297116 +0000 UTC m=+418.049513482" lastFinishedPulling="2026-04-16 16:36:52.986106423 +0000 UTC m=+422.383322791" observedRunningTime="2026-04-16 16:36:53.628006308 +0000 UTC m=+423.025222696" watchObservedRunningTime="2026-04-16 16:36:53.628024748 +0000 UTC m=+423.025241135"
Apr 16 16:36:56.624170 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:56.624137    2578 generic.go:358] "Generic (PLEG): container finished" podID="b8d03003-d679-4684-b987-75a4ec1a678f" containerID="8139e77a4377315359b62146d40ad79dc51be81ee078e8de2165c98e8c533ea5" exitCode=0
Apr 16 16:36:56.624629 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:56.624209    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-wxvfp" event={"ID":"b8d03003-d679-4684-b987-75a4ec1a678f","Type":"ContainerDied","Data":"8139e77a4377315359b62146d40ad79dc51be81ee078e8de2165c98e8c533ea5"}
Apr 16 16:36:57.743589 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:57.743569    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-wxvfp"
Apr 16 16:36:57.851677 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:57.851653    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nsx8\" (UniqueName: \"kubernetes.io/projected/b8d03003-d679-4684-b987-75a4ec1a678f-kube-api-access-5nsx8\") pod \"b8d03003-d679-4684-b987-75a4ec1a678f\" (UID: \"b8d03003-d679-4684-b987-75a4ec1a678f\") "
Apr 16 16:36:57.853587 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:57.853565    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d03003-d679-4684-b987-75a4ec1a678f-kube-api-access-5nsx8" (OuterVolumeSpecName: "kube-api-access-5nsx8") pod "b8d03003-d679-4684-b987-75a4ec1a678f" (UID: "b8d03003-d679-4684-b987-75a4ec1a678f"). InnerVolumeSpecName "kube-api-access-5nsx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:36:57.874346 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:57.874322    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d44c76d5d-74zpg"]
Apr 16 16:36:57.874626 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:57.874614    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d03003-d679-4684-b987-75a4ec1a678f" containerName="s3-init"
Apr 16 16:36:57.874674 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:57.874627    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d03003-d679-4684-b987-75a4ec1a678f" containerName="s3-init"
Apr 16 16:36:57.874708 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:57.874677    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="b8d03003-d679-4684-b987-75a4ec1a678f" containerName="s3-init"
Apr 16 16:36:57.877634 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:57.877619    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:57.885769 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:57.885750    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d44c76d5d-74zpg"]
Apr 16 16:36:57.952705 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:57.952682    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nsx8\" (UniqueName: \"kubernetes.io/projected/b8d03003-d679-4684-b987-75a4ec1a678f-kube-api-access-5nsx8\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:36:58.057732 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.057709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-console-serving-cert\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.057852 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.057771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-oauth-serving-cert\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.057852 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.057792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mj72\" (UniqueName: \"kubernetes.io/projected/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-kube-api-access-7mj72\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.057852 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.057818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-console-oauth-config\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.058041 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.057907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-service-ca\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.058041 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.057940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-trusted-ca-bundle\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.058041 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.057970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-console-config\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.159092 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.159039    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-oauth-serving-cert\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.159092 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.159066    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7mj72\" (UniqueName: \"kubernetes.io/projected/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-kube-api-access-7mj72\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.159092 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.159086    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-console-oauth-config\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.159302 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.159116    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-service-ca\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.159302 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.159132    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-trusted-ca-bundle\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.159302 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.159159    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-console-config\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.159302 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.159188    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-console-serving-cert\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.159841 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.159818    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-service-ca\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.159972 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.159951    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-console-config\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.160023 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.159970    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-oauth-serving-cert\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.160252 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.160229    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-trusted-ca-bundle\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.161631 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.161609    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-console-oauth-config\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.161715 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.161636    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-console-serving-cert\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.165968 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.165952    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mj72\" (UniqueName: \"kubernetes.io/projected/89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208-kube-api-access-7mj72\") pod \"console-5d44c76d5d-74zpg\" (UID: \"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208\") " pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.185847 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.185830    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:36:58.298205 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.298183    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d44c76d5d-74zpg"]
Apr 16 16:36:58.300781 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:36:58.300752    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e9c8a0_9fc5_4d46_aaa4_bcdbe6257208.slice/crio-f5468095b763d2bc9aaf04fd1748bb5c02d3197374800908c8d9c56cd0129ec7 WatchSource:0}: Error finding container f5468095b763d2bc9aaf04fd1748bb5c02d3197374800908c8d9c56cd0129ec7: Status 404 returned error can't find the container with id f5468095b763d2bc9aaf04fd1748bb5c02d3197374800908c8d9c56cd0129ec7
Apr 16 16:36:58.631441 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.631409    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d44c76d5d-74zpg" event={"ID":"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208","Type":"ContainerStarted","Data":"d11a957e12ceaf7d8f35f4769a573dc6b5f41b8f9be8009ec3be34064bc6d67b"}
Apr 16 16:36:58.631441 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.631444    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d44c76d5d-74zpg" event={"ID":"89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208","Type":"ContainerStarted","Data":"f5468095b763d2bc9aaf04fd1748bb5c02d3197374800908c8d9c56cd0129ec7"}
Apr 16 16:36:58.632514 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.632494    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-wxvfp" event={"ID":"b8d03003-d679-4684-b987-75a4ec1a678f","Type":"ContainerDied","Data":"fd0a4bbea34479fcafadbf02ceb0244cf2501d39e98325c0786db05feaf1ee78"}
Apr 16 16:36:58.632514 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.632517    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd0a4bbea34479fcafadbf02ceb0244cf2501d39e98325c0786db05feaf1ee78"
Apr 16 16:36:58.632662 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.632541    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-wxvfp"
Apr 16 16:36:58.647184 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:58.647146    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5d44c76d5d-74zpg" podStartSLOduration=1.647136489 podStartE2EDuration="1.647136489s" podCreationTimestamp="2026-04-16 16:36:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:36:58.646247649 +0000 UTC m=+428.043464049" watchObservedRunningTime="2026-04-16 16:36:58.647136489 +0000 UTC m=+428.044352876"
Apr 16 16:36:59.328596 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:59.328512    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6"]
Apr 16 16:36:59.331719 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:59.331701    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6"
Apr 16 16:36:59.333916 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:59.333888    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Apr 16 16:36:59.333916 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:59.333913    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-x4ndd\""
Apr 16 16:36:59.337546 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:59.337525    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6"]
Apr 16 16:36:59.467691 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:59.467667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw2q7\" (UniqueName: \"kubernetes.io/projected/cb2c2a03-6f1e-4204-b906-010e3009a128-kube-api-access-qw2q7\") pod \"seaweedfs-tls-custom-ddd4dbfd-jpph6\" (UID: \"cb2c2a03-6f1e-4204-b906-010e3009a128\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6"
Apr 16 16:36:59.467816 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:59.467739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cb2c2a03-6f1e-4204-b906-010e3009a128-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-jpph6\" (UID: \"cb2c2a03-6f1e-4204-b906-010e3009a128\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6"
Apr 16 16:36:59.568524 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:59.568500    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cb2c2a03-6f1e-4204-b906-010e3009a128-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-jpph6\" (UID: \"cb2c2a03-6f1e-4204-b906-010e3009a128\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6"
Apr 16 16:36:59.568628 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:59.568548    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qw2q7\" (UniqueName: \"kubernetes.io/projected/cb2c2a03-6f1e-4204-b906-010e3009a128-kube-api-access-qw2q7\") pod \"seaweedfs-tls-custom-ddd4dbfd-jpph6\" (UID: \"cb2c2a03-6f1e-4204-b906-010e3009a128\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6"
Apr 16 16:36:59.568858 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:59.568842    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cb2c2a03-6f1e-4204-b906-010e3009a128-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-jpph6\" (UID: \"cb2c2a03-6f1e-4204-b906-010e3009a128\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6"
Apr 16 16:36:59.575630 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:59.575611    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw2q7\" (UniqueName: \"kubernetes.io/projected/cb2c2a03-6f1e-4204-b906-010e3009a128-kube-api-access-qw2q7\") pod \"seaweedfs-tls-custom-ddd4dbfd-jpph6\" (UID: \"cb2c2a03-6f1e-4204-b906-010e3009a128\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6"
Apr 16 16:36:59.642001 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:59.641950    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6"
Apr 16 16:36:59.755789 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:36:59.755769    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6"]
Apr 16 16:36:59.757963 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:36:59.757936    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2c2a03_6f1e_4204_b906_010e3009a128.slice/crio-03f1792299de61743204ee7fa71b1cfdc18b3fee155be5cdbaa490503cc64e28 WatchSource:0}: Error finding container 03f1792299de61743204ee7fa71b1cfdc18b3fee155be5cdbaa490503cc64e28: Status 404 returned error can't find the container with id 03f1792299de61743204ee7fa71b1cfdc18b3fee155be5cdbaa490503cc64e28
Apr 16 16:37:00.641017 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:00.640980    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6" event={"ID":"cb2c2a03-6f1e-4204-b906-010e3009a128","Type":"ContainerStarted","Data":"03f1792299de61743204ee7fa71b1cfdc18b3fee155be5cdbaa490503cc64e28"}
Apr 16 16:37:02.648542 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:02.648461    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6" event={"ID":"cb2c2a03-6f1e-4204-b906-010e3009a128","Type":"ContainerStarted","Data":"06e7d64ddde315234aba972aefaa2bb52f98b287105e41a217a2b1eb53c3c4df"}
Apr 16 16:37:02.663072 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:02.663028    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6" podStartSLOduration=1.201148258 podStartE2EDuration="3.663017178s" podCreationTimestamp="2026-04-16 16:36:59 +0000 UTC" firstStartedPulling="2026-04-16 16:36:59.759240561 +0000 UTC m=+429.156456927" lastFinishedPulling="2026-04-16 16:37:02.22110948 +0000 UTC m=+431.618325847" observedRunningTime="2026-04-16 16:37:02.661958219 +0000 UTC m=+432.059174608" watchObservedRunningTime="2026-04-16 16:37:02.663017178 +0000 UTC m=+432.060233569"
Apr 16 16:37:04.208302 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:04.208267    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6"]
Apr 16 16:37:04.655361 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:04.655264    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6" podUID="cb2c2a03-6f1e-4204-b906-010e3009a128" containerName="seaweedfs-tls-custom" containerID="cri-o://06e7d64ddde315234aba972aefaa2bb52f98b287105e41a217a2b1eb53c3c4df" gracePeriod=30
Apr 16 16:37:05.884948 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:05.884925    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6"
Apr 16 16:37:05.916836 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:05.916773    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw2q7\" (UniqueName: \"kubernetes.io/projected/cb2c2a03-6f1e-4204-b906-010e3009a128-kube-api-access-qw2q7\") pod \"cb2c2a03-6f1e-4204-b906-010e3009a128\" (UID: \"cb2c2a03-6f1e-4204-b906-010e3009a128\") "
Apr 16 16:37:05.916951 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:05.916858    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cb2c2a03-6f1e-4204-b906-010e3009a128-data\") pod \"cb2c2a03-6f1e-4204-b906-010e3009a128\" (UID: \"cb2c2a03-6f1e-4204-b906-010e3009a128\") "
Apr 16 16:37:05.918053 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:05.918026    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2c2a03-6f1e-4204-b906-010e3009a128-data" (OuterVolumeSpecName: "data") pod "cb2c2a03-6f1e-4204-b906-010e3009a128" (UID: "cb2c2a03-6f1e-4204-b906-010e3009a128"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:37:05.918764 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:05.918738    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2c2a03-6f1e-4204-b906-010e3009a128-kube-api-access-qw2q7" (OuterVolumeSpecName: "kube-api-access-qw2q7") pod "cb2c2a03-6f1e-4204-b906-010e3009a128" (UID: "cb2c2a03-6f1e-4204-b906-010e3009a128"). InnerVolumeSpecName "kube-api-access-qw2q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:37:06.018392 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.018369    2578 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cb2c2a03-6f1e-4204-b906-010e3009a128-data\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:37:06.018392 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.018391    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw2q7\" (UniqueName: \"kubernetes.io/projected/cb2c2a03-6f1e-4204-b906-010e3009a128-kube-api-access-qw2q7\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:37:06.661925 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.661894    2578 generic.go:358] "Generic (PLEG): container finished" podID="cb2c2a03-6f1e-4204-b906-010e3009a128" containerID="06e7d64ddde315234aba972aefaa2bb52f98b287105e41a217a2b1eb53c3c4df" exitCode=0
Apr 16 16:37:06.662102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.661951    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6"
Apr 16 16:37:06.662102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.661973    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6" event={"ID":"cb2c2a03-6f1e-4204-b906-010e3009a128","Type":"ContainerDied","Data":"06e7d64ddde315234aba972aefaa2bb52f98b287105e41a217a2b1eb53c3c4df"}
Apr 16 16:37:06.662102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.662007    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6" event={"ID":"cb2c2a03-6f1e-4204-b906-010e3009a128","Type":"ContainerDied","Data":"03f1792299de61743204ee7fa71b1cfdc18b3fee155be5cdbaa490503cc64e28"}
Apr 16 16:37:06.662102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.662022    2578 scope.go:117] "RemoveContainer" containerID="06e7d64ddde315234aba972aefaa2bb52f98b287105e41a217a2b1eb53c3c4df"
Apr 16 16:37:06.670480 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.670464    2578 scope.go:117] "RemoveContainer" containerID="06e7d64ddde315234aba972aefaa2bb52f98b287105e41a217a2b1eb53c3c4df"
Apr 16 16:37:06.670748 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:37:06.670732    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e7d64ddde315234aba972aefaa2bb52f98b287105e41a217a2b1eb53c3c4df\": container with ID starting with 06e7d64ddde315234aba972aefaa2bb52f98b287105e41a217a2b1eb53c3c4df not found: ID does not exist" containerID="06e7d64ddde315234aba972aefaa2bb52f98b287105e41a217a2b1eb53c3c4df"
Apr 16 16:37:06.670810 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.670757    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e7d64ddde315234aba972aefaa2bb52f98b287105e41a217a2b1eb53c3c4df"} err="failed to get container status \"06e7d64ddde315234aba972aefaa2bb52f98b287105e41a217a2b1eb53c3c4df\": rpc error: code = NotFound desc = could not find container \"06e7d64ddde315234aba972aefaa2bb52f98b287105e41a217a2b1eb53c3c4df\": container with ID starting with 06e7d64ddde315234aba972aefaa2bb52f98b287105e41a217a2b1eb53c3c4df not found: ID does not exist"
Apr 16 16:37:06.682284 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.682263    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6"]
Apr 16 16:37:06.685289 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.685270    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-jpph6"]
Apr 16 16:37:06.714986 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.714962    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv"]
Apr 16 16:37:06.715276 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.715264    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb2c2a03-6f1e-4204-b906-010e3009a128" containerName="seaweedfs-tls-custom"
Apr 16 16:37:06.715318 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.715277    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb2c2a03-6f1e-4204-b906-010e3009a128" containerName="seaweedfs-tls-custom"
Apr 16 16:37:06.715354 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.715328    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="cb2c2a03-6f1e-4204-b906-010e3009a128" containerName="seaweedfs-tls-custom"
Apr 16 16:37:06.719478 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.719464    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv"
Apr 16 16:37:06.721849 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.721823    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Apr 16 16:37:06.721849 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.721847    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-x4ndd\""
Apr 16 16:37:06.721999 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.721827    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Apr 16 16:37:06.727639 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.725022    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv"]
Apr 16 16:37:06.823586 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.823546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mphm2\" (UniqueName: \"kubernetes.io/projected/627c8c08-25bd-4e9f-9b1b-1baf6c101d30-kube-api-access-mphm2\") pod \"seaweedfs-tls-custom-5c88b85bb7-4dfxv\" (UID: \"627c8c08-25bd-4e9f-9b1b-1baf6c101d30\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv"
Apr 16 16:37:06.823693 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.823605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/627c8c08-25bd-4e9f-9b1b-1baf6c101d30-data\") pod \"seaweedfs-tls-custom-5c88b85bb7-4dfxv\" (UID: \"627c8c08-25bd-4e9f-9b1b-1baf6c101d30\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv"
Apr 16 16:37:06.823693 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.823643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/627c8c08-25bd-4e9f-9b1b-1baf6c101d30-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-5c88b85bb7-4dfxv\" (UID: \"627c8c08-25bd-4e9f-9b1b-1baf6c101d30\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv"
Apr 16 16:37:06.924831 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.924766    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mphm2\" (UniqueName: \"kubernetes.io/projected/627c8c08-25bd-4e9f-9b1b-1baf6c101d30-kube-api-access-mphm2\") pod \"seaweedfs-tls-custom-5c88b85bb7-4dfxv\" (UID: \"627c8c08-25bd-4e9f-9b1b-1baf6c101d30\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv"
Apr 16 16:37:06.924831 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.924816    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/627c8c08-25bd-4e9f-9b1b-1baf6c101d30-data\") pod \"seaweedfs-tls-custom-5c88b85bb7-4dfxv\" (UID: \"627c8c08-25bd-4e9f-9b1b-1baf6c101d30\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv"
Apr 16 16:37:06.925140 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.924854    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/627c8c08-25bd-4e9f-9b1b-1baf6c101d30-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-5c88b85bb7-4dfxv\" (UID: \"627c8c08-25bd-4e9f-9b1b-1baf6c101d30\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv"
Apr 16 16:37:06.925176 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.925156    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/627c8c08-25bd-4e9f-9b1b-1baf6c101d30-data\") pod \"seaweedfs-tls-custom-5c88b85bb7-4dfxv\" (UID: \"627c8c08-25bd-4e9f-9b1b-1baf6c101d30\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv"
Apr 16 16:37:06.927132 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.927116    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/627c8c08-25bd-4e9f-9b1b-1baf6c101d30-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-5c88b85bb7-4dfxv\" (UID: \"627c8c08-25bd-4e9f-9b1b-1baf6c101d30\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv"
Apr 16 16:37:06.931936 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:06.931913    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mphm2\" (UniqueName: \"kubernetes.io/projected/627c8c08-25bd-4e9f-9b1b-1baf6c101d30-kube-api-access-mphm2\") pod \"seaweedfs-tls-custom-5c88b85bb7-4dfxv\" (UID: \"627c8c08-25bd-4e9f-9b1b-1baf6c101d30\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv"
Apr 16 16:37:07.033392 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:07.033369    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv"
Apr 16 16:37:07.146281 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:07.146253    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv"]
Apr 16 16:37:07.148946 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:37:07.148918    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627c8c08_25bd_4e9f_9b1b_1baf6c101d30.slice/crio-d1bc26ff506ff1d989d88e57e88c7bc347e6491ff04c9c7c59b1307a806cf281 WatchSource:0}: Error finding container d1bc26ff506ff1d989d88e57e88c7bc347e6491ff04c9c7c59b1307a806cf281: Status 404 returned error can't find the container with id d1bc26ff506ff1d989d88e57e88c7bc347e6491ff04c9c7c59b1307a806cf281
Apr 16 16:37:07.234959 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:07.234933    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb2c2a03-6f1e-4204-b906-010e3009a128" path="/var/lib/kubelet/pods/cb2c2a03-6f1e-4204-b906-010e3009a128/volumes"
Apr 16 16:37:07.666748 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:07.666677    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv" event={"ID":"627c8c08-25bd-4e9f-9b1b-1baf6c101d30","Type":"ContainerStarted","Data":"3297ac16672d539c601b40ab0b3f24fb562812a13bb0ac43bc9834a5e898540f"}
Apr 16 16:37:07.666748 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:07.666709    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv" event={"ID":"627c8c08-25bd-4e9f-9b1b-1baf6c101d30","Type":"ContainerStarted","Data":"d1bc26ff506ff1d989d88e57e88c7bc347e6491ff04c9c7c59b1307a806cf281"}
Apr 16 16:37:07.683478 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:07.683436    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-4dfxv" podStartSLOduration=1.410508606 podStartE2EDuration="1.683419556s" podCreationTimestamp="2026-04-16 16:37:06 +0000 UTC" firstStartedPulling="2026-04-16 16:37:07.150275042 +0000 UTC m=+436.547491409" lastFinishedPulling="2026-04-16 16:37:07.423185975 +0000 UTC m=+436.820402359" observedRunningTime="2026-04-16 16:37:07.682101881 +0000 UTC m=+437.079318260" watchObservedRunningTime="2026-04-16 16:37:07.683419556 +0000 UTC m=+437.080635944"
Apr 16 16:37:07.966984 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:07.966958    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-bb56r"]
Apr 16 16:37:07.970412 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:07.970394    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-bb56r"
Apr 16 16:37:07.975786 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:07.975766    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-bb56r"]
Apr 16 16:37:08.033861 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:08.033841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w855b\" (UniqueName: \"kubernetes.io/projected/4db2d7a6-8e21-4163-bae8-85e0aad131a5-kube-api-access-w855b\") pod \"s3-tls-init-custom-bb56r\" (UID: \"4db2d7a6-8e21-4163-bae8-85e0aad131a5\") " pod="kserve/s3-tls-init-custom-bb56r"
Apr 16 16:37:08.135214 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:08.135188    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w855b\" (UniqueName: \"kubernetes.io/projected/4db2d7a6-8e21-4163-bae8-85e0aad131a5-kube-api-access-w855b\") pod \"s3-tls-init-custom-bb56r\" (UID: \"4db2d7a6-8e21-4163-bae8-85e0aad131a5\") " pod="kserve/s3-tls-init-custom-bb56r"
Apr 16 16:37:08.143271 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:08.143252    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w855b\" (UniqueName: \"kubernetes.io/projected/4db2d7a6-8e21-4163-bae8-85e0aad131a5-kube-api-access-w855b\") pod \"s3-tls-init-custom-bb56r\" (UID: \"4db2d7a6-8e21-4163-bae8-85e0aad131a5\") " pod="kserve/s3-tls-init-custom-bb56r"
Apr 16 16:37:08.186597 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:08.186573    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:37:08.186677 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:08.186606    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:37:08.191052 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:08.191032    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:37:08.289931 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:08.289869    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-bb56r"
Apr 16 16:37:08.415577 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:08.415524    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-bb56r"]
Apr 16 16:37:08.418456 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:37:08.418430    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4db2d7a6_8e21_4163_bae8_85e0aad131a5.slice/crio-b67232e9b59d78659d3ff88707c52d854473f38e699d1faf074cfac508bc15d4 WatchSource:0}: Error finding container b67232e9b59d78659d3ff88707c52d854473f38e699d1faf074cfac508bc15d4: Status 404 returned error can't find the container with id b67232e9b59d78659d3ff88707c52d854473f38e699d1faf074cfac508bc15d4
Apr 16 16:37:08.671297 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:08.671208    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-bb56r" event={"ID":"4db2d7a6-8e21-4163-bae8-85e0aad131a5","Type":"ContainerStarted","Data":"ffa469ccbd51db99e694284abc54870091fa0bf22eefcd7c8a05d958df6fe974"}
Apr 16 16:37:08.671297 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:08.671251    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-bb56r" event={"ID":"4db2d7a6-8e21-4163-bae8-85e0aad131a5","Type":"ContainerStarted","Data":"b67232e9b59d78659d3ff88707c52d854473f38e699d1faf074cfac508bc15d4"}
Apr 16 16:37:08.675080 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:08.675059    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d44c76d5d-74zpg"
Apr 16 16:37:08.685973 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:08.685928    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-bb56r" podStartSLOduration=1.685910383 podStartE2EDuration="1.685910383s" podCreationTimestamp="2026-04-16 16:37:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:37:08.684022047 +0000 UTC m=+438.081238435" watchObservedRunningTime="2026-04-16 16:37:08.685910383 +0000 UTC m=+438.083126772"
Apr 16 16:37:08.726190 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:08.726162    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64cc66665-ndb47"]
Apr 16 16:37:12.682878 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:12.682850    2578 generic.go:358] "Generic (PLEG): container finished" podID="4db2d7a6-8e21-4163-bae8-85e0aad131a5" containerID="ffa469ccbd51db99e694284abc54870091fa0bf22eefcd7c8a05d958df6fe974" exitCode=0
Apr 16 16:37:12.683165 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:12.682915    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-bb56r" event={"ID":"4db2d7a6-8e21-4163-bae8-85e0aad131a5","Type":"ContainerDied","Data":"ffa469ccbd51db99e694284abc54870091fa0bf22eefcd7c8a05d958df6fe974"}
Apr 16 16:37:13.799885 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:13.799863    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-bb56r"
Apr 16 16:37:13.881204 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:13.881176    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w855b\" (UniqueName: \"kubernetes.io/projected/4db2d7a6-8e21-4163-bae8-85e0aad131a5-kube-api-access-w855b\") pod \"4db2d7a6-8e21-4163-bae8-85e0aad131a5\" (UID: \"4db2d7a6-8e21-4163-bae8-85e0aad131a5\") "
Apr 16 16:37:13.883100 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:13.883069    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db2d7a6-8e21-4163-bae8-85e0aad131a5-kube-api-access-w855b" (OuterVolumeSpecName: "kube-api-access-w855b") pod "4db2d7a6-8e21-4163-bae8-85e0aad131a5" (UID: "4db2d7a6-8e21-4163-bae8-85e0aad131a5"). InnerVolumeSpecName "kube-api-access-w855b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:37:13.982679 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:13.982660    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w855b\" (UniqueName: \"kubernetes.io/projected/4db2d7a6-8e21-4163-bae8-85e0aad131a5-kube-api-access-w855b\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:37:14.689580 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:14.689548    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-bb56r"
Apr 16 16:37:14.689580 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:14.689544    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-bb56r" event={"ID":"4db2d7a6-8e21-4163-bae8-85e0aad131a5","Type":"ContainerDied","Data":"b67232e9b59d78659d3ff88707c52d854473f38e699d1faf074cfac508bc15d4"}
Apr 16 16:37:14.689755 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:14.689586    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b67232e9b59d78659d3ff88707c52d854473f38e699d1faf074cfac508bc15d4"
Apr 16 16:37:16.690570 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:16.690520    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-4pgdf"]
Apr 16 16:37:16.690993 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:16.690966    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db2d7a6-8e21-4163-bae8-85e0aad131a5" containerName="s3-tls-init-custom"
Apr 16 16:37:16.690993 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:16.690983    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db2d7a6-8e21-4163-bae8-85e0aad131a5" containerName="s3-tls-init-custom"
Apr 16 16:37:16.691102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:16.691067    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="4db2d7a6-8e21-4163-bae8-85e0aad131a5" containerName="s3-tls-init-custom"
Apr 16 16:37:16.693590 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:16.693549    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4pgdf"
Apr 16 16:37:16.695780 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:16.695760    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Apr 16 16:37:16.702892 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:16.701521    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-4pgdf"]
Apr 16 16:37:16.801480 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:16.801455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rss54\" (UniqueName: \"kubernetes.io/projected/9926b7e5-4866-41e3-b468-3459dd74ea36-kube-api-access-rss54\") pod \"s3-tls-init-serving-4pgdf\" (UID: \"9926b7e5-4866-41e3-b468-3459dd74ea36\") " pod="kserve/s3-tls-init-serving-4pgdf"
Apr 16 16:37:16.902317 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:16.902292    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rss54\" (UniqueName: \"kubernetes.io/projected/9926b7e5-4866-41e3-b468-3459dd74ea36-kube-api-access-rss54\") pod \"s3-tls-init-serving-4pgdf\" (UID: \"9926b7e5-4866-41e3-b468-3459dd74ea36\") " pod="kserve/s3-tls-init-serving-4pgdf"
Apr 16 16:37:16.910480 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:16.910455    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rss54\" (UniqueName: \"kubernetes.io/projected/9926b7e5-4866-41e3-b468-3459dd74ea36-kube-api-access-rss54\") pod \"s3-tls-init-serving-4pgdf\" (UID: \"9926b7e5-4866-41e3-b468-3459dd74ea36\") " pod="kserve/s3-tls-init-serving-4pgdf"
Apr 16 16:37:17.022773 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:17.022748    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4pgdf"
Apr 16 16:37:17.140405 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:17.140377    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-4pgdf"]
Apr 16 16:37:17.142311 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:37:17.142279    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9926b7e5_4866_41e3_b468_3459dd74ea36.slice/crio-e8c87170860efffb5da9a171ec6247c33b43c39dc3755453814f9fd5c55476ea WatchSource:0}: Error finding container e8c87170860efffb5da9a171ec6247c33b43c39dc3755453814f9fd5c55476ea: Status 404 returned error can't find the container with id e8c87170860efffb5da9a171ec6247c33b43c39dc3755453814f9fd5c55476ea
Apr 16 16:37:17.702163 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:17.702128    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4pgdf" event={"ID":"9926b7e5-4866-41e3-b468-3459dd74ea36","Type":"ContainerStarted","Data":"a2ebad2302f61d6b34938779524811b9dd2c0f8577e9c872e824af315c60e50e"}
Apr 16 16:37:17.702163 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:17.702163    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4pgdf" event={"ID":"9926b7e5-4866-41e3-b468-3459dd74ea36","Type":"ContainerStarted","Data":"e8c87170860efffb5da9a171ec6247c33b43c39dc3755453814f9fd5c55476ea"}
Apr 16 16:37:17.716801 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:17.716755    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-4pgdf" podStartSLOduration=1.716743031 podStartE2EDuration="1.716743031s" podCreationTimestamp="2026-04-16 16:37:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:37:17.714843507 +0000 UTC m=+447.112059892" watchObservedRunningTime="2026-04-16 16:37:17.716743031 +0000 UTC m=+447.113959415"
Apr 16 16:37:23.720649 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:23.720617    2578 generic.go:358] "Generic (PLEG): container finished" podID="9926b7e5-4866-41e3-b468-3459dd74ea36" containerID="a2ebad2302f61d6b34938779524811b9dd2c0f8577e9c872e824af315c60e50e" exitCode=0
Apr 16 16:37:23.721025 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:23.720690    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4pgdf" event={"ID":"9926b7e5-4866-41e3-b468-3459dd74ea36","Type":"ContainerDied","Data":"a2ebad2302f61d6b34938779524811b9dd2c0f8577e9c872e824af315c60e50e"}
Apr 16 16:37:24.841293 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:24.841268    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4pgdf"
Apr 16 16:37:24.963645 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:24.963623    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rss54\" (UniqueName: \"kubernetes.io/projected/9926b7e5-4866-41e3-b468-3459dd74ea36-kube-api-access-rss54\") pod \"9926b7e5-4866-41e3-b468-3459dd74ea36\" (UID: \"9926b7e5-4866-41e3-b468-3459dd74ea36\") "
Apr 16 16:37:24.965647 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:24.965623    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9926b7e5-4866-41e3-b468-3459dd74ea36-kube-api-access-rss54" (OuterVolumeSpecName: "kube-api-access-rss54") pod "9926b7e5-4866-41e3-b468-3459dd74ea36" (UID: "9926b7e5-4866-41e3-b468-3459dd74ea36"). InnerVolumeSpecName "kube-api-access-rss54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:37:25.064140 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:25.064087    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rss54\" (UniqueName: \"kubernetes.io/projected/9926b7e5-4866-41e3-b468-3459dd74ea36-kube-api-access-rss54\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:37:25.727296 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:25.727266    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4pgdf" event={"ID":"9926b7e5-4866-41e3-b468-3459dd74ea36","Type":"ContainerDied","Data":"e8c87170860efffb5da9a171ec6247c33b43c39dc3755453814f9fd5c55476ea"}
Apr 16 16:37:25.727296 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:25.727295    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8c87170860efffb5da9a171ec6247c33b43c39dc3755453814f9fd5c55476ea"
Apr 16 16:37:25.727469 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:25.727302    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4pgdf"
Apr 16 16:37:33.153565 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:33.153520    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"]
Apr 16 16:37:33.153928 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:33.153808    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9926b7e5-4866-41e3-b468-3459dd74ea36" containerName="s3-tls-init-serving"
Apr 16 16:37:33.153928 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:33.153819    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9926b7e5-4866-41e3-b468-3459dd74ea36" containerName="s3-tls-init-serving"
Apr 16 16:37:33.153928 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:33.153901    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="9926b7e5-4866-41e3-b468-3459dd74ea36" containerName="s3-tls-init-serving"
Apr 16 16:37:33.157794 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:33.157778    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"
Apr 16 16:37:33.159896 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:33.159864    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fhlr7\""
Apr 16 16:37:33.163710 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:33.163681    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"]
Apr 16 16:37:33.220820 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:33.220787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cc3509d-c830-4e32-877b-2e9d258ecdff-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8\" (UID: \"6cc3509d-c830-4e32-877b-2e9d258ecdff\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"
Apr 16 16:37:33.321712 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:33.321689    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cc3509d-c830-4e32-877b-2e9d258ecdff-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8\" (UID: \"6cc3509d-c830-4e32-877b-2e9d258ecdff\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"
Apr 16 16:37:33.322049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:33.322030    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cc3509d-c830-4e32-877b-2e9d258ecdff-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8\" (UID: \"6cc3509d-c830-4e32-877b-2e9d258ecdff\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"
Apr 16 16:37:33.468244 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:33.468221    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"
Apr 16 16:37:33.583134 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:33.583103    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"]
Apr 16 16:37:33.585958 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:37:33.585922    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc3509d_c830_4e32_877b_2e9d258ecdff.slice/crio-44eef0143b14b128bca106843418196172e021f4118ad11138d603f24798672c WatchSource:0}: Error finding container 44eef0143b14b128bca106843418196172e021f4118ad11138d603f24798672c: Status 404 returned error can't find the container with id 44eef0143b14b128bca106843418196172e021f4118ad11138d603f24798672c
Apr 16 16:37:33.746314 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:33.746240    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-64cc66665-ndb47" podUID="bad3ae4b-579c-4f82-a6b2-16416b11f28e" containerName="console" containerID="cri-o://e2296c9aae1209ddcf2e444dce9e589a68f06156328b5e54201a0c2ec6b63553" gracePeriod=15
Apr 16 16:37:33.751122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:33.751026    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" event={"ID":"6cc3509d-c830-4e32-877b-2e9d258ecdff","Type":"ContainerStarted","Data":"44eef0143b14b128bca106843418196172e021f4118ad11138d603f24798672c"}
Apr 16 16:37:33.973780 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:33.973761    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-64cc66665-ndb47_bad3ae4b-579c-4f82-a6b2-16416b11f28e/console/0.log"
Apr 16 16:37:33.973877 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:33.973820    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:37:34.027098 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.027036    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-trusted-ca-bundle\") pod \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") "
Apr 16 16:37:34.027218 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.027100    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-oauth-config\") pod \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") "
Apr 16 16:37:34.027218 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.027125    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-serving-cert\") pod \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") "
Apr 16 16:37:34.027218 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.027161    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgr4z\" (UniqueName: \"kubernetes.io/projected/bad3ae4b-579c-4f82-a6b2-16416b11f28e-kube-api-access-pgr4z\") pod \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") "
Apr 16 16:37:34.027218 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.027191    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-config\") pod \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") "
Apr 16 16:37:34.027504 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.027335    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-oauth-serving-cert\") pod \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") "
Apr 16 16:37:34.027641 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.027509    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bad3ae4b-579c-4f82-a6b2-16416b11f28e" (UID: "bad3ae4b-579c-4f82-a6b2-16416b11f28e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:37:34.027641 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.027596    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-config" (OuterVolumeSpecName: "console-config") pod "bad3ae4b-579c-4f82-a6b2-16416b11f28e" (UID: "bad3ae4b-579c-4f82-a6b2-16416b11f28e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:37:34.027758 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.027660    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-service-ca\") pod \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\" (UID: \"bad3ae4b-579c-4f82-a6b2-16416b11f28e\") "
Apr 16 16:37:34.027945 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.027922    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-service-ca" (OuterVolumeSpecName: "service-ca") pod "bad3ae4b-579c-4f82-a6b2-16416b11f28e" (UID: "bad3ae4b-579c-4f82-a6b2-16416b11f28e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:37:34.027987 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.027935    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-trusted-ca-bundle\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:37:34.027987 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.027977    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-config\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:37:34.028056 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.027988    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bad3ae4b-579c-4f82-a6b2-16416b11f28e" (UID: "bad3ae4b-579c-4f82-a6b2-16416b11f28e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:37:34.029606 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.029586    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bad3ae4b-579c-4f82-a6b2-16416b11f28e" (UID: "bad3ae4b-579c-4f82-a6b2-16416b11f28e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:37:34.029830 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.029811    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bad3ae4b-579c-4f82-a6b2-16416b11f28e" (UID: "bad3ae4b-579c-4f82-a6b2-16416b11f28e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:37:34.029916 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.029898    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad3ae4b-579c-4f82-a6b2-16416b11f28e-kube-api-access-pgr4z" (OuterVolumeSpecName: "kube-api-access-pgr4z") pod "bad3ae4b-579c-4f82-a6b2-16416b11f28e" (UID: "bad3ae4b-579c-4f82-a6b2-16416b11f28e"). InnerVolumeSpecName "kube-api-access-pgr4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:37:34.128425 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.128401    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-service-ca\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:37:34.128425 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.128422    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-oauth-config\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:37:34.128545 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.128432    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bad3ae4b-579c-4f82-a6b2-16416b11f28e-console-serving-cert\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:37:34.128545 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.128441    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgr4z\" (UniqueName: \"kubernetes.io/projected/bad3ae4b-579c-4f82-a6b2-16416b11f28e-kube-api-access-pgr4z\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:37:34.128545 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.128449    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bad3ae4b-579c-4f82-a6b2-16416b11f28e-oauth-serving-cert\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:37:34.755567 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.755476    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-64cc66665-ndb47_bad3ae4b-579c-4f82-a6b2-16416b11f28e/console/0.log"
Apr 16 16:37:34.755567 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.755529    2578 generic.go:358] "Generic (PLEG): container finished" podID="bad3ae4b-579c-4f82-a6b2-16416b11f28e" containerID="e2296c9aae1209ddcf2e444dce9e589a68f06156328b5e54201a0c2ec6b63553" exitCode=2
Apr 16 16:37:34.756062 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.755586    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cc66665-ndb47" event={"ID":"bad3ae4b-579c-4f82-a6b2-16416b11f28e","Type":"ContainerDied","Data":"e2296c9aae1209ddcf2e444dce9e589a68f06156328b5e54201a0c2ec6b63553"}
Apr 16 16:37:34.756062 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.755622    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cc66665-ndb47"
Apr 16 16:37:34.756062 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.755638    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cc66665-ndb47" event={"ID":"bad3ae4b-579c-4f82-a6b2-16416b11f28e","Type":"ContainerDied","Data":"f18bbdc286cd878c960eee972bd52e897e1a6a98c0fb90917df885f8a6c0d65f"}
Apr 16 16:37:34.756062 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.755659    2578 scope.go:117] "RemoveContainer" containerID="e2296c9aae1209ddcf2e444dce9e589a68f06156328b5e54201a0c2ec6b63553"
Apr 16 16:37:34.765878 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.765798    2578 scope.go:117] "RemoveContainer" containerID="e2296c9aae1209ddcf2e444dce9e589a68f06156328b5e54201a0c2ec6b63553"
Apr 16 16:37:34.766206 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:37:34.766179    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2296c9aae1209ddcf2e444dce9e589a68f06156328b5e54201a0c2ec6b63553\": container with ID starting with e2296c9aae1209ddcf2e444dce9e589a68f06156328b5e54201a0c2ec6b63553 not found: ID does not exist" containerID="e2296c9aae1209ddcf2e444dce9e589a68f06156328b5e54201a0c2ec6b63553"
Apr 16 16:37:34.766383 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.766219    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2296c9aae1209ddcf2e444dce9e589a68f06156328b5e54201a0c2ec6b63553"} err="failed to get container status \"e2296c9aae1209ddcf2e444dce9e589a68f06156328b5e54201a0c2ec6b63553\": rpc error: code = NotFound desc = could not find container \"e2296c9aae1209ddcf2e444dce9e589a68f06156328b5e54201a0c2ec6b63553\": container with ID starting with e2296c9aae1209ddcf2e444dce9e589a68f06156328b5e54201a0c2ec6b63553 not found: ID does not exist"
Apr 16 16:37:34.779737 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.779699    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64cc66665-ndb47"]
Apr 16 16:37:34.782469 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:34.782442    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64cc66665-ndb47"]
Apr 16 16:37:35.235538 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:35.235512    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bad3ae4b-579c-4f82-a6b2-16416b11f28e" path="/var/lib/kubelet/pods/bad3ae4b-579c-4f82-a6b2-16416b11f28e/volumes"
Apr 16 16:37:36.764129 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:36.764094    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" event={"ID":"6cc3509d-c830-4e32-877b-2e9d258ecdff","Type":"ContainerStarted","Data":"5ed74b1fe2e9568e63691986b1452fcf96ebba79ea0f97029c7265cd285c1696"}
Apr 16 16:37:40.776679 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:40.776651    2578 generic.go:358] "Generic (PLEG): container finished" podID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerID="5ed74b1fe2e9568e63691986b1452fcf96ebba79ea0f97029c7265cd285c1696" exitCode=0
Apr 16 16:37:40.777066 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:40.776730    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" event={"ID":"6cc3509d-c830-4e32-877b-2e9d258ecdff","Type":"ContainerDied","Data":"5ed74b1fe2e9568e63691986b1452fcf96ebba79ea0f97029c7265cd285c1696"}
Apr 16 16:37:53.824794 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:53.824764    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" event={"ID":"6cc3509d-c830-4e32-877b-2e9d258ecdff","Type":"ContainerStarted","Data":"48034a3a87d2323c79db1cfc54f2d592c3985d5f374e0efc711450bf25a1a238"}
Apr 16 16:37:56.834780 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:56.834745    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" event={"ID":"6cc3509d-c830-4e32-877b-2e9d258ecdff","Type":"ContainerStarted","Data":"b9cb28788e82c7009b642c2d311644c1bccb68bb5b66f2460e9b5544e946f42b"}
Apr 16 16:37:56.835184 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:56.835012    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"
Apr 16 16:37:56.836101 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:56.836073    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Apr 16 16:37:56.850468 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:56.850432    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podStartSLOduration=1.4278518980000001 podStartE2EDuration="23.850420611s" podCreationTimestamp="2026-04-16 16:37:33 +0000 UTC" firstStartedPulling="2026-04-16 16:37:33.588010391 +0000 UTC m=+462.985226763" lastFinishedPulling="2026-04-16 16:37:56.010579089 +0000 UTC m=+485.407795476" observedRunningTime="2026-04-16 16:37:56.849006325 +0000 UTC m=+486.246222710" watchObservedRunningTime="2026-04-16 16:37:56.850420611 +0000 UTC m=+486.247637000"
Apr 16 16:37:57.838279 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:57.838250    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"
Apr 16 16:37:57.838758 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:57.838363    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Apr 16 16:37:57.839319 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:57.839295    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:37:58.841313 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:58.841276    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Apr 16 16:37:58.841718 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:37:58.841572    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:38:08.841545 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:38:08.841499    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Apr 16 16:38:08.842052 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:38:08.841944    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:38:18.841310 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:38:18.841273    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Apr 16 16:38:18.841825 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:38:18.841799    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:38:28.841440 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:38:28.841392    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Apr 16 16:38:28.841965 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:38:28.841917    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:38:38.841358 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:38:38.841270    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Apr 16 16:38:38.841771 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:38:38.841747    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:38:48.842042 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:38:48.841993    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Apr 16 16:38:48.842478 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:38:48.842453    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:38:58.841708 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:38:58.841679    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"
Apr 16 16:38:58.842195 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:38:58.842012    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"
Apr 16 16:39:08.372935 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:08.372895    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"]
Apr 16 16:39:08.373452 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:08.373219    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="kserve-container" containerID="cri-o://48034a3a87d2323c79db1cfc54f2d592c3985d5f374e0efc711450bf25a1a238" gracePeriod=30
Apr 16 16:39:08.373452 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:08.373305    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="agent" containerID="cri-o://b9cb28788e82c7009b642c2d311644c1bccb68bb5b66f2460e9b5544e946f42b" gracePeriod=30
Apr 16 16:39:08.452125 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:08.452098    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"]
Apr 16 16:39:08.452405 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:08.452393    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad3ae4b-579c-4f82-a6b2-16416b11f28e" containerName="console"
Apr 16 16:39:08.452448 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:08.452406    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad3ae4b-579c-4f82-a6b2-16416b11f28e" containerName="console"
Apr 16 16:39:08.452482 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:08.452468    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="bad3ae4b-579c-4f82-a6b2-16416b11f28e" containerName="console"
Apr 16 16:39:08.454594 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:08.454576    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"
Apr 16 16:39:08.461775 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:08.461752    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"]
Apr 16 16:39:08.526887 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:08.526864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl\" (UID: \"ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"
Apr 16 16:39:08.627352 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:08.627295    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl\" (UID: \"ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"
Apr 16 16:39:08.627608 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:08.627591    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl\" (UID: \"ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"
Apr 16 16:39:08.765397 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:08.765370    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"
Apr 16 16:39:08.842122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:08.842064    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Apr 16 16:39:08.842462 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:08.842432    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:39:08.879444 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:08.879411    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"]
Apr 16 16:39:08.881997 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:39:08.881971    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddce7cdb_a86e_4b0c_9e7d_7c932fa6d150.slice/crio-b70623f8a27131b0ee81c6f27b63bac7619dc9b540678e537f4e69655641f26a WatchSource:0}: Error finding container b70623f8a27131b0ee81c6f27b63bac7619dc9b540678e537f4e69655641f26a: Status 404 returned error can't find the container with id b70623f8a27131b0ee81c6f27b63bac7619dc9b540678e537f4e69655641f26a
Apr 16 16:39:09.049361 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:09.049325    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" event={"ID":"ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150","Type":"ContainerStarted","Data":"a9b5504a31872d274698d09acd21f86e24914fceadbbccdb32364bc0f7a071b4"}
Apr 16 16:39:09.049501 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:09.049368    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" event={"ID":"ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150","Type":"ContainerStarted","Data":"b70623f8a27131b0ee81c6f27b63bac7619dc9b540678e537f4e69655641f26a"}
Apr 16 16:39:12.059430 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:12.059403    2578 generic.go:358] "Generic (PLEG): container finished" podID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerID="48034a3a87d2323c79db1cfc54f2d592c3985d5f374e0efc711450bf25a1a238" exitCode=0
Apr 16 16:39:12.059755 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:12.059475    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" event={"ID":"6cc3509d-c830-4e32-877b-2e9d258ecdff","Type":"ContainerDied","Data":"48034a3a87d2323c79db1cfc54f2d592c3985d5f374e0efc711450bf25a1a238"}
Apr 16 16:39:13.063022 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:13.062988    2578 generic.go:358] "Generic (PLEG): container finished" podID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerID="a9b5504a31872d274698d09acd21f86e24914fceadbbccdb32364bc0f7a071b4" exitCode=0
Apr 16 16:39:13.063360 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:13.063062    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" event={"ID":"ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150","Type":"ContainerDied","Data":"a9b5504a31872d274698d09acd21f86e24914fceadbbccdb32364bc0f7a071b4"}
Apr 16 16:39:14.067223 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:14.067190    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" event={"ID":"ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150","Type":"ContainerStarted","Data":"15f45f5322097ea7b4ddbcc01bb1a52a50f33101907cd0ddb05f4a4bdbec15a8"}
Apr 16 16:39:14.067613 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:14.067229    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" event={"ID":"ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150","Type":"ContainerStarted","Data":"7d09f561a3235d5759ccb8e8a1999a3a9a3cfd3f2249833b60b7a47fb9073b36"}
Apr 16 16:39:14.067613 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:14.067525    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"
Apr 16 16:39:14.068856 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:14.068827    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Apr 16 16:39:14.084064 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:14.084027    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podStartSLOduration=6.084015442 podStartE2EDuration="6.084015442s" podCreationTimestamp="2026-04-16 16:39:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:39:14.080944823 +0000 UTC m=+563.478161211" watchObservedRunningTime="2026-04-16 16:39:14.084015442 +0000 UTC m=+563.481231827"
Apr 16 16:39:15.070460 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:15.070431    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"
Apr 16 16:39:15.070828 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:15.070532    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Apr 16 16:39:15.071488 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:15.071461    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:39:16.073635 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:16.073595    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Apr 16 16:39:16.074081 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:16.074057    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:39:18.841703 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:18.841661    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Apr 16 16:39:18.842319 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:18.842295    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:39:26.073680 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:26.073626    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Apr 16 16:39:26.074090 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:26.074052    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:39:28.841403 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:28.841343    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Apr 16 16:39:28.841844 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:28.841489    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"
Apr 16 16:39:28.841844 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:28.841696    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:39:28.841844 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:28.841788    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"
Apr 16 16:39:36.073812 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:36.073762    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Apr 16 16:39:36.074209 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:36.074188    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:39:38.502351 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:38.502330    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"
Apr 16 16:39:38.639803 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:38.639739    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cc3509d-c830-4e32-877b-2e9d258ecdff-kserve-provision-location\") pod \"6cc3509d-c830-4e32-877b-2e9d258ecdff\" (UID: \"6cc3509d-c830-4e32-877b-2e9d258ecdff\") "
Apr 16 16:39:38.640090 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:38.640068    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc3509d-c830-4e32-877b-2e9d258ecdff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6cc3509d-c830-4e32-877b-2e9d258ecdff" (UID: "6cc3509d-c830-4e32-877b-2e9d258ecdff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:39:38.740810 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:38.740789    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cc3509d-c830-4e32-877b-2e9d258ecdff-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:39:39.150363 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:39.150322    2578 generic.go:358] "Generic (PLEG): container finished" podID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerID="b9cb28788e82c7009b642c2d311644c1bccb68bb5b66f2460e9b5544e946f42b" exitCode=0
Apr 16 16:39:39.150511 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:39.150394    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"
Apr 16 16:39:39.150511 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:39.150409    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" event={"ID":"6cc3509d-c830-4e32-877b-2e9d258ecdff","Type":"ContainerDied","Data":"b9cb28788e82c7009b642c2d311644c1bccb68bb5b66f2460e9b5544e946f42b"}
Apr 16 16:39:39.150511 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:39.150450    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8" event={"ID":"6cc3509d-c830-4e32-877b-2e9d258ecdff","Type":"ContainerDied","Data":"44eef0143b14b128bca106843418196172e021f4118ad11138d603f24798672c"}
Apr 16 16:39:39.150511 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:39.150469    2578 scope.go:117] "RemoveContainer" containerID="b9cb28788e82c7009b642c2d311644c1bccb68bb5b66f2460e9b5544e946f42b"
Apr 16 16:39:39.158840 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:39.158824    2578 scope.go:117] "RemoveContainer" containerID="48034a3a87d2323c79db1cfc54f2d592c3985d5f374e0efc711450bf25a1a238"
Apr 16 16:39:39.165677 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:39.165662    2578 scope.go:117] "RemoveContainer" containerID="5ed74b1fe2e9568e63691986b1452fcf96ebba79ea0f97029c7265cd285c1696"
Apr 16 16:39:39.171593 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:39.171572    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"]
Apr 16 16:39:39.172506 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:39.172490    2578 scope.go:117] "RemoveContainer" containerID="b9cb28788e82c7009b642c2d311644c1bccb68bb5b66f2460e9b5544e946f42b"
Apr 16 16:39:39.172763 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:39:39.172745    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9cb28788e82c7009b642c2d311644c1bccb68bb5b66f2460e9b5544e946f42b\": container with ID starting with b9cb28788e82c7009b642c2d311644c1bccb68bb5b66f2460e9b5544e946f42b not found: ID does not exist" containerID="b9cb28788e82c7009b642c2d311644c1bccb68bb5b66f2460e9b5544e946f42b"
Apr 16 16:39:39.172813 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:39.172773    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9cb28788e82c7009b642c2d311644c1bccb68bb5b66f2460e9b5544e946f42b"} err="failed to get container status \"b9cb28788e82c7009b642c2d311644c1bccb68bb5b66f2460e9b5544e946f42b\": rpc error: code = NotFound desc = could not find container \"b9cb28788e82c7009b642c2d311644c1bccb68bb5b66f2460e9b5544e946f42b\": container with ID starting with b9cb28788e82c7009b642c2d311644c1bccb68bb5b66f2460e9b5544e946f42b not found: ID does not exist"
Apr 16 16:39:39.172813 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:39.172790    2578 scope.go:117] "RemoveContainer" containerID="48034a3a87d2323c79db1cfc54f2d592c3985d5f374e0efc711450bf25a1a238"
Apr 16 16:39:39.173081 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:39:39.173065    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48034a3a87d2323c79db1cfc54f2d592c3985d5f374e0efc711450bf25a1a238\": container with ID starting with 48034a3a87d2323c79db1cfc54f2d592c3985d5f374e0efc711450bf25a1a238 not found: ID does not exist" containerID="48034a3a87d2323c79db1cfc54f2d592c3985d5f374e0efc711450bf25a1a238"
Apr 16 16:39:39.173122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:39.173085    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48034a3a87d2323c79db1cfc54f2d592c3985d5f374e0efc711450bf25a1a238"} err="failed to get container status \"48034a3a87d2323c79db1cfc54f2d592c3985d5f374e0efc711450bf25a1a238\": rpc error: code = NotFound desc = could not find container \"48034a3a87d2323c79db1cfc54f2d592c3985d5f374e0efc711450bf25a1a238\": container with ID starting with 48034a3a87d2323c79db1cfc54f2d592c3985d5f374e0efc711450bf25a1a238 not found: ID does not exist"
Apr 16 16:39:39.173122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:39.173100    2578 scope.go:117] "RemoveContainer" containerID="5ed74b1fe2e9568e63691986b1452fcf96ebba79ea0f97029c7265cd285c1696"
Apr 16 16:39:39.173389 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:39:39.173350    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed74b1fe2e9568e63691986b1452fcf96ebba79ea0f97029c7265cd285c1696\": container with ID starting with 5ed74b1fe2e9568e63691986b1452fcf96ebba79ea0f97029c7265cd285c1696 not found: ID does not exist" containerID="5ed74b1fe2e9568e63691986b1452fcf96ebba79ea0f97029c7265cd285c1696"
Apr 16 16:39:39.173487 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:39.173386    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed74b1fe2e9568e63691986b1452fcf96ebba79ea0f97029c7265cd285c1696"} err="failed to get container status \"5ed74b1fe2e9568e63691986b1452fcf96ebba79ea0f97029c7265cd285c1696\": rpc error: code = NotFound desc = could not find container \"5ed74b1fe2e9568e63691986b1452fcf96ebba79ea0f97029c7265cd285c1696\": container with ID starting with 5ed74b1fe2e9568e63691986b1452fcf96ebba79ea0f97029c7265cd285c1696 not found: ID does not exist"
Apr 16 16:39:39.175516 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:39.175499    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f85f4fcc6-rsxs8"]
Apr 16 16:39:39.234647 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:39.234625    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" path="/var/lib/kubelet/pods/6cc3509d-c830-4e32-877b-2e9d258ecdff/volumes"
Apr 16 16:39:46.074161 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:46.074120    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Apr 16 16:39:46.074641 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:46.074537    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:39:51.146781 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:51.146752    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 16:39:51.147689 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:51.147664    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 16:39:56.073903 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:56.073860    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Apr 16 16:39:56.074295 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:39:56.074259    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:40:06.074219 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:06.074122    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Apr 16 16:40:06.074640 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:06.074611    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:40:16.074746 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:16.074717    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"
Apr 16 16:40:16.075200 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:16.075101    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"
Apr 16 16:40:23.516726 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:23.516695    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"]
Apr 16 16:40:23.517303 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:23.516991    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="kserve-container" containerID="cri-o://7d09f561a3235d5759ccb8e8a1999a3a9a3cfd3f2249833b60b7a47fb9073b36" gracePeriod=30
Apr 16 16:40:23.517303 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:23.517052    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="agent" containerID="cri-o://15f45f5322097ea7b4ddbcc01bb1a52a50f33101907cd0ddb05f4a4bdbec15a8" gracePeriod=30
Apr 16 16:40:26.074120 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:26.074078    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Apr 16 16:40:26.075656 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:26.075623    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:40:27.293003 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:27.292974    2578 generic.go:358] "Generic (PLEG): container finished" podID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerID="7d09f561a3235d5759ccb8e8a1999a3a9a3cfd3f2249833b60b7a47fb9073b36" exitCode=0
Apr 16 16:40:27.293316 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:27.293022    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" event={"ID":"ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150","Type":"ContainerDied","Data":"7d09f561a3235d5759ccb8e8a1999a3a9a3cfd3f2249833b60b7a47fb9073b36"}
Apr 16 16:40:33.613355 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:33.613317    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"]
Apr 16 16:40:33.613873 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:33.613798    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="storage-initializer"
Apr 16 16:40:33.613873 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:33.613817    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="storage-initializer"
Apr 16 16:40:33.613873 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:33.613854    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="agent"
Apr 16 16:40:33.613873 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:33.613861    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="agent"
Apr 16 16:40:33.614086 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:33.613876    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="kserve-container"
Apr 16 16:40:33.614086 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:33.613885    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="kserve-container"
Apr 16 16:40:33.614086 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:33.613951    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="agent"
Apr 16 16:40:33.614086 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:33.613969    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="6cc3509d-c830-4e32-877b-2e9d258ecdff" containerName="kserve-container"
Apr 16 16:40:33.616804 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:33.616783    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"
Apr 16 16:40:33.622799 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:33.622775    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"]
Apr 16 16:40:33.719501 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:33.719477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3148437-c0b1-44d3-9fea-3f7254ee765d-kserve-provision-location\") pod \"isvc-logger-predictor-67b779c5c9-g7v2j\" (UID: \"a3148437-c0b1-44d3-9fea-3f7254ee765d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"
Apr 16 16:40:33.820153 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:33.820128    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3148437-c0b1-44d3-9fea-3f7254ee765d-kserve-provision-location\") pod \"isvc-logger-predictor-67b779c5c9-g7v2j\" (UID: \"a3148437-c0b1-44d3-9fea-3f7254ee765d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"
Apr 16 16:40:33.820432 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:33.820418    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3148437-c0b1-44d3-9fea-3f7254ee765d-kserve-provision-location\") pod \"isvc-logger-predictor-67b779c5c9-g7v2j\" (UID: \"a3148437-c0b1-44d3-9fea-3f7254ee765d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"
Apr 16 16:40:33.927022 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:33.926970    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"
Apr 16 16:40:34.040892 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:34.040864    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"]
Apr 16 16:40:34.043888 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:40:34.043861    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3148437_c0b1_44d3_9fea_3f7254ee765d.slice/crio-44997ec032ccac9e260620435b977974252273ff0e2a9d75da60115ac7e92037 WatchSource:0}: Error finding container 44997ec032ccac9e260620435b977974252273ff0e2a9d75da60115ac7e92037: Status 404 returned error can't find the container with id 44997ec032ccac9e260620435b977974252273ff0e2a9d75da60115ac7e92037
Apr 16 16:40:34.313298 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:34.313261    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" event={"ID":"a3148437-c0b1-44d3-9fea-3f7254ee765d","Type":"ContainerStarted","Data":"c988f546f20bc9871eeda375c0f55e19d339c59f913c760c24f244eb3b673823"}
Apr 16 16:40:34.313434 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:34.313307    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" event={"ID":"a3148437-c0b1-44d3-9fea-3f7254ee765d","Type":"ContainerStarted","Data":"44997ec032ccac9e260620435b977974252273ff0e2a9d75da60115ac7e92037"}
Apr 16 16:40:36.074171 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:36.074134    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Apr 16 16:40:36.075693 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:36.075669    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:40:38.325265 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:38.325194    2578 generic.go:358] "Generic (PLEG): container finished" podID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerID="c988f546f20bc9871eeda375c0f55e19d339c59f913c760c24f244eb3b673823" exitCode=0
Apr 16 16:40:38.325265 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:38.325260    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" event={"ID":"a3148437-c0b1-44d3-9fea-3f7254ee765d","Type":"ContainerDied","Data":"c988f546f20bc9871eeda375c0f55e19d339c59f913c760c24f244eb3b673823"}
Apr 16 16:40:39.330630 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:39.330583    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" event={"ID":"a3148437-c0b1-44d3-9fea-3f7254ee765d","Type":"ContainerStarted","Data":"fe64283fb28a7c4bbf80f26a9964e7396fc3e400013b61a8d9df97b838688772"}
Apr 16 16:40:39.330630 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:39.330623    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" event={"ID":"a3148437-c0b1-44d3-9fea-3f7254ee765d","Type":"ContainerStarted","Data":"569bb16c1d3532958dec65ee4d0b1e20c04c88ed540558b3d51a09153327818c"}
Apr 16 16:40:39.331265 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:39.330875    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"
Apr 16 16:40:39.332231 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:39.332204    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Apr 16 16:40:39.345974 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:39.345936    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podStartSLOduration=6.345926315 podStartE2EDuration="6.345926315s" podCreationTimestamp="2026-04-16 16:40:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:40:39.344984672 +0000 UTC m=+648.742201060" watchObservedRunningTime="2026-04-16 16:40:39.345926315 +0000 UTC m=+648.743142702"
Apr 16 16:40:40.334303 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:40.334266    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"
Apr 16 16:40:40.334739 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:40.334418    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Apr 16 16:40:40.335602 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:40.335575    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:40:41.337266 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:41.337227    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Apr 16 16:40:41.337853 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:41.337826    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:40:46.073877 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:46.073831    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Apr 16 16:40:46.074284 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:46.073966    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"
Apr 16 16:40:46.076462 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:46.076436    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:40:46.076564 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:46.076538    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"
Apr 16 16:40:51.337991 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:51.337934    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Apr 16 16:40:51.338479 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:51.338456    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:40:53.658953 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:53.658932    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"
Apr 16 16:40:53.765671 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:53.765647    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150-kserve-provision-location\") pod \"ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150\" (UID: \"ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150\") "
Apr 16 16:40:53.765931 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:53.765910    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" (UID: "ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:40:53.867026 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:53.866974    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:40:54.372695 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:54.372666    2578 generic.go:358] "Generic (PLEG): container finished" podID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerID="15f45f5322097ea7b4ddbcc01bb1a52a50f33101907cd0ddb05f4a4bdbec15a8" exitCode=0
Apr 16 16:40:54.372900 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:54.372709    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" event={"ID":"ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150","Type":"ContainerDied","Data":"15f45f5322097ea7b4ddbcc01bb1a52a50f33101907cd0ddb05f4a4bdbec15a8"}
Apr 16 16:40:54.372900 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:54.372755    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl" event={"ID":"ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150","Type":"ContainerDied","Data":"b70623f8a27131b0ee81c6f27b63bac7619dc9b540678e537f4e69655641f26a"}
Apr 16 16:40:54.372900 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:54.372756    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"
Apr 16 16:40:54.372900 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:54.372770    2578 scope.go:117] "RemoveContainer" containerID="15f45f5322097ea7b4ddbcc01bb1a52a50f33101907cd0ddb05f4a4bdbec15a8"
Apr 16 16:40:54.383422 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:54.383405    2578 scope.go:117] "RemoveContainer" containerID="7d09f561a3235d5759ccb8e8a1999a3a9a3cfd3f2249833b60b7a47fb9073b36"
Apr 16 16:40:54.390182 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:54.390163    2578 scope.go:117] "RemoveContainer" containerID="a9b5504a31872d274698d09acd21f86e24914fceadbbccdb32364bc0f7a071b4"
Apr 16 16:40:54.394032 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:54.394014    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"]
Apr 16 16:40:54.397053 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:54.397035    2578 scope.go:117] "RemoveContainer" containerID="15f45f5322097ea7b4ddbcc01bb1a52a50f33101907cd0ddb05f4a4bdbec15a8"
Apr 16 16:40:54.397341 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:40:54.397312    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f45f5322097ea7b4ddbcc01bb1a52a50f33101907cd0ddb05f4a4bdbec15a8\": container with ID starting with 15f45f5322097ea7b4ddbcc01bb1a52a50f33101907cd0ddb05f4a4bdbec15a8 not found: ID does not exist" containerID="15f45f5322097ea7b4ddbcc01bb1a52a50f33101907cd0ddb05f4a4bdbec15a8"
Apr 16 16:40:54.397453 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:54.397350    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f45f5322097ea7b4ddbcc01bb1a52a50f33101907cd0ddb05f4a4bdbec15a8"} err="failed to get container status \"15f45f5322097ea7b4ddbcc01bb1a52a50f33101907cd0ddb05f4a4bdbec15a8\": rpc error: code = NotFound desc = could not find container \"15f45f5322097ea7b4ddbcc01bb1a52a50f33101907cd0ddb05f4a4bdbec15a8\": container with ID starting with 15f45f5322097ea7b4ddbcc01bb1a52a50f33101907cd0ddb05f4a4bdbec15a8 not found: ID does not exist"
Apr 16 16:40:54.397453 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:54.397375    2578 scope.go:117] "RemoveContainer" containerID="7d09f561a3235d5759ccb8e8a1999a3a9a3cfd3f2249833b60b7a47fb9073b36"
Apr 16 16:40:54.397995 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:40:54.397970    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d09f561a3235d5759ccb8e8a1999a3a9a3cfd3f2249833b60b7a47fb9073b36\": container with ID starting with 7d09f561a3235d5759ccb8e8a1999a3a9a3cfd3f2249833b60b7a47fb9073b36 not found: ID does not exist" containerID="7d09f561a3235d5759ccb8e8a1999a3a9a3cfd3f2249833b60b7a47fb9073b36"
Apr 16 16:40:54.398077 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:54.398002    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d09f561a3235d5759ccb8e8a1999a3a9a3cfd3f2249833b60b7a47fb9073b36"} err="failed to get container status \"7d09f561a3235d5759ccb8e8a1999a3a9a3cfd3f2249833b60b7a47fb9073b36\": rpc error: code = NotFound desc = could not find container \"7d09f561a3235d5759ccb8e8a1999a3a9a3cfd3f2249833b60b7a47fb9073b36\": container with ID starting with 7d09f561a3235d5759ccb8e8a1999a3a9a3cfd3f2249833b60b7a47fb9073b36 not found: ID does not exist"
Apr 16 16:40:54.398077 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:54.398024    2578 scope.go:117] "RemoveContainer" containerID="a9b5504a31872d274698d09acd21f86e24914fceadbbccdb32364bc0f7a071b4"
Apr 16 16:40:54.398294 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:40:54.398273    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b5504a31872d274698d09acd21f86e24914fceadbbccdb32364bc0f7a071b4\": container with ID starting with a9b5504a31872d274698d09acd21f86e24914fceadbbccdb32364bc0f7a071b4 not found: ID does not exist" containerID="a9b5504a31872d274698d09acd21f86e24914fceadbbccdb32364bc0f7a071b4"
Apr 16 16:40:54.398347 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:54.398301    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b5504a31872d274698d09acd21f86e24914fceadbbccdb32364bc0f7a071b4"} err="failed to get container status \"a9b5504a31872d274698d09acd21f86e24914fceadbbccdb32364bc0f7a071b4\": rpc error: code = NotFound desc = could not find container \"a9b5504a31872d274698d09acd21f86e24914fceadbbccdb32364bc0f7a071b4\": container with ID starting with a9b5504a31872d274698d09acd21f86e24914fceadbbccdb32364bc0f7a071b4 not found: ID does not exist"
Apr 16 16:40:54.398962 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:54.398944    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-746b56d97c-pl6nl"]
Apr 16 16:40:55.235613 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:40:55.235586    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" path="/var/lib/kubelet/pods/ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150/volumes"
Apr 16 16:41:01.337844 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:01.337804    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Apr 16 16:41:01.338296 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:01.338283    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:41:11.338013 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:11.337977    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Apr 16 16:41:11.338547 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:11.338515    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:41:21.338073 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:21.338035    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Apr 16 16:41:21.338522 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:21.338493    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:41:31.337458 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:31.337376    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Apr 16 16:41:31.337895 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:31.337871    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:41:41.338331 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:41.338302    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"
Apr 16 16:41:41.338747 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:41.338607    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"
Apr 16 16:41:48.820816 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:48.820786    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"]
Apr 16 16:41:48.821368 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:48.821084    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="kserve-container" containerID="cri-o://569bb16c1d3532958dec65ee4d0b1e20c04c88ed540558b3d51a09153327818c" gracePeriod=30
Apr 16 16:41:48.821368 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:48.821168    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="agent" containerID="cri-o://fe64283fb28a7c4bbf80f26a9964e7396fc3e400013b61a8d9df97b838688772" gracePeriod=30
Apr 16 16:41:48.849195 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:48.849169    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq"]
Apr 16 16:41:48.849497 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:48.849474    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="kserve-container"
Apr 16 16:41:48.849497 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:48.849487    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="kserve-container"
Apr 16 16:41:48.849592 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:48.849499    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="storage-initializer"
Apr 16 16:41:48.849592 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:48.849506    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="storage-initializer"
Apr 16 16:41:48.849592 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:48.849518    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="agent"
Apr 16 16:41:48.849592 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:48.849524    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="agent"
Apr 16 16:41:48.849592 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:48.849587    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="agent"
Apr 16 16:41:48.849738 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:48.849595    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="ddce7cdb-a86e-4b0c-9e7d-7c932fa6d150" containerName="kserve-container"
Apr 16 16:41:48.852675 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:48.852661    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq"
Apr 16 16:41:48.859583 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:48.859543    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq"]
Apr 16 16:41:48.953965 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:48.953940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de8da7ab-283e-409d-b9c3-e4a35ece25e3-kserve-provision-location\") pod \"isvc-lightgbm-predictor-78c8d484d6-dzzdq\" (UID: \"de8da7ab-283e-409d-b9c3-e4a35ece25e3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq"
Apr 16 16:41:49.054277 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:49.054251    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de8da7ab-283e-409d-b9c3-e4a35ece25e3-kserve-provision-location\") pod \"isvc-lightgbm-predictor-78c8d484d6-dzzdq\" (UID: \"de8da7ab-283e-409d-b9c3-e4a35ece25e3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq"
Apr 16 16:41:49.054632 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:49.054614    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de8da7ab-283e-409d-b9c3-e4a35ece25e3-kserve-provision-location\") pod \"isvc-lightgbm-predictor-78c8d484d6-dzzdq\" (UID: \"de8da7ab-283e-409d-b9c3-e4a35ece25e3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq"
Apr 16 16:41:49.163686 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:49.163619    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq"
Apr 16 16:41:49.277376 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:49.277347    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq"]
Apr 16 16:41:49.280122 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:41:49.280095    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8da7ab_283e_409d_b9c3_e4a35ece25e3.slice/crio-cba18e347f107795e35d4e2d2dc1dbeb315033e99cb12ebba40a16a0fad2f939 WatchSource:0}: Error finding container cba18e347f107795e35d4e2d2dc1dbeb315033e99cb12ebba40a16a0fad2f939: Status 404 returned error can't find the container with id cba18e347f107795e35d4e2d2dc1dbeb315033e99cb12ebba40a16a0fad2f939
Apr 16 16:41:49.281932 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:49.281914    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 16:41:49.515226 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:49.515197    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" event={"ID":"de8da7ab-283e-409d-b9c3-e4a35ece25e3","Type":"ContainerStarted","Data":"632ade3a10944edf1afe0b50b2286e76b1c86048f98f4665757fdda7676bc4bd"}
Apr 16 16:41:49.515226 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:49.515229    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" event={"ID":"de8da7ab-283e-409d-b9c3-e4a35ece25e3","Type":"ContainerStarted","Data":"cba18e347f107795e35d4e2d2dc1dbeb315033e99cb12ebba40a16a0fad2f939"}
Apr 16 16:41:51.337448 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:51.337410    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Apr 16 16:41:51.337902 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:51.337767    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:41:52.524502 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:52.524428    2578 generic.go:358] "Generic (PLEG): container finished" podID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerID="569bb16c1d3532958dec65ee4d0b1e20c04c88ed540558b3d51a09153327818c" exitCode=0
Apr 16 16:41:52.524818 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:52.524503    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" event={"ID":"a3148437-c0b1-44d3-9fea-3f7254ee765d","Type":"ContainerDied","Data":"569bb16c1d3532958dec65ee4d0b1e20c04c88ed540558b3d51a09153327818c"}
Apr 16 16:41:53.528065 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:53.528034    2578 generic.go:358] "Generic (PLEG): container finished" podID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerID="632ade3a10944edf1afe0b50b2286e76b1c86048f98f4665757fdda7676bc4bd" exitCode=0
Apr 16 16:41:53.528345 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:41:53.528106    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" event={"ID":"de8da7ab-283e-409d-b9c3-e4a35ece25e3","Type":"ContainerDied","Data":"632ade3a10944edf1afe0b50b2286e76b1c86048f98f4665757fdda7676bc4bd"}
Apr 16 16:42:00.554035 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:00.554000    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" event={"ID":"de8da7ab-283e-409d-b9c3-e4a35ece25e3","Type":"ContainerStarted","Data":"bb9d5961dabbf9977d7d99e16d5a7a53851e2e61dd953c4d7edd7282e9ba2ff9"}
Apr 16 16:42:00.554429 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:00.554399    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq"
Apr 16 16:42:00.555458 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:00.555430    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" podUID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Apr 16 16:42:00.569929 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:00.569894    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" podStartSLOduration=6.54928091 podStartE2EDuration="12.56988225s" podCreationTimestamp="2026-04-16 16:41:48 +0000 UTC" firstStartedPulling="2026-04-16 16:41:53.529173714 +0000 UTC m=+722.926390080" lastFinishedPulling="2026-04-16 16:41:59.549775042 +0000 UTC m=+728.946991420" observedRunningTime="2026-04-16 16:42:00.568590431 +0000 UTC m=+729.965806819" watchObservedRunningTime="2026-04-16 16:42:00.56988225 +0000 UTC m=+729.967098700"
Apr 16 16:42:01.337654 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:01.337616    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Apr 16 16:42:01.337924 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:01.337893    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:42:01.556695 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:01.556662    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" podUID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Apr 16 16:42:11.338170 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:11.338122    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Apr 16 16:42:11.338598 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:11.338243    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"
Apr 16 16:42:11.338598 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:11.338429    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:42:11.338598 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:11.338517    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"
Apr 16 16:42:11.557300 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:11.557267    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" podUID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Apr 16 16:42:18.995373 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:18.995349    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"
Apr 16 16:42:19.088240 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.088160    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3148437-c0b1-44d3-9fea-3f7254ee765d-kserve-provision-location\") pod \"a3148437-c0b1-44d3-9fea-3f7254ee765d\" (UID: \"a3148437-c0b1-44d3-9fea-3f7254ee765d\") "
Apr 16 16:42:19.088481 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.088461    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3148437-c0b1-44d3-9fea-3f7254ee765d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a3148437-c0b1-44d3-9fea-3f7254ee765d" (UID: "a3148437-c0b1-44d3-9fea-3f7254ee765d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:42:19.189111 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.189086    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3148437-c0b1-44d3-9fea-3f7254ee765d-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:42:19.606977 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.606946    2578 generic.go:358] "Generic (PLEG): container finished" podID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerID="fe64283fb28a7c4bbf80f26a9964e7396fc3e400013b61a8d9df97b838688772" exitCode=137
Apr 16 16:42:19.607149 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.607015    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"
Apr 16 16:42:19.607149 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.607018    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" event={"ID":"a3148437-c0b1-44d3-9fea-3f7254ee765d","Type":"ContainerDied","Data":"fe64283fb28a7c4bbf80f26a9964e7396fc3e400013b61a8d9df97b838688772"}
Apr 16 16:42:19.607149 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.607062    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j" event={"ID":"a3148437-c0b1-44d3-9fea-3f7254ee765d","Type":"ContainerDied","Data":"44997ec032ccac9e260620435b977974252273ff0e2a9d75da60115ac7e92037"}
Apr 16 16:42:19.607149 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.607082    2578 scope.go:117] "RemoveContainer" containerID="fe64283fb28a7c4bbf80f26a9964e7396fc3e400013b61a8d9df97b838688772"
Apr 16 16:42:19.614408 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.614393    2578 scope.go:117] "RemoveContainer" containerID="569bb16c1d3532958dec65ee4d0b1e20c04c88ed540558b3d51a09153327818c"
Apr 16 16:42:19.621403 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.621375    2578 scope.go:117] "RemoveContainer" containerID="c988f546f20bc9871eeda375c0f55e19d339c59f913c760c24f244eb3b673823"
Apr 16 16:42:19.622342 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.622327    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"]
Apr 16 16:42:19.627737 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.627717    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-67b779c5c9-g7v2j"]
Apr 16 16:42:19.628484 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.628470    2578 scope.go:117] "RemoveContainer" containerID="fe64283fb28a7c4bbf80f26a9964e7396fc3e400013b61a8d9df97b838688772"
Apr 16 16:42:19.628776 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:42:19.628756    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe64283fb28a7c4bbf80f26a9964e7396fc3e400013b61a8d9df97b838688772\": container with ID starting with fe64283fb28a7c4bbf80f26a9964e7396fc3e400013b61a8d9df97b838688772 not found: ID does not exist" containerID="fe64283fb28a7c4bbf80f26a9964e7396fc3e400013b61a8d9df97b838688772"
Apr 16 16:42:19.628861 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.628783    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe64283fb28a7c4bbf80f26a9964e7396fc3e400013b61a8d9df97b838688772"} err="failed to get container status \"fe64283fb28a7c4bbf80f26a9964e7396fc3e400013b61a8d9df97b838688772\": rpc error: code = NotFound desc = could not find container \"fe64283fb28a7c4bbf80f26a9964e7396fc3e400013b61a8d9df97b838688772\": container with ID starting with fe64283fb28a7c4bbf80f26a9964e7396fc3e400013b61a8d9df97b838688772 not found: ID does not exist"
Apr 16 16:42:19.628861 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.628800    2578 scope.go:117] "RemoveContainer" containerID="569bb16c1d3532958dec65ee4d0b1e20c04c88ed540558b3d51a09153327818c"
Apr 16 16:42:19.629027 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:42:19.629010    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"569bb16c1d3532958dec65ee4d0b1e20c04c88ed540558b3d51a09153327818c\": container with ID starting with 569bb16c1d3532958dec65ee4d0b1e20c04c88ed540558b3d51a09153327818c not found: ID does not exist" containerID="569bb16c1d3532958dec65ee4d0b1e20c04c88ed540558b3d51a09153327818c"
Apr 16 16:42:19.629067 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.629033    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"569bb16c1d3532958dec65ee4d0b1e20c04c88ed540558b3d51a09153327818c"} err="failed to get container status \"569bb16c1d3532958dec65ee4d0b1e20c04c88ed540558b3d51a09153327818c\": rpc error: code = NotFound desc = could not find container \"569bb16c1d3532958dec65ee4d0b1e20c04c88ed540558b3d51a09153327818c\": container with ID starting with 569bb16c1d3532958dec65ee4d0b1e20c04c88ed540558b3d51a09153327818c not found: ID does not exist"
Apr 16 16:42:19.629067 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.629048    2578 scope.go:117] "RemoveContainer" containerID="c988f546f20bc9871eeda375c0f55e19d339c59f913c760c24f244eb3b673823"
Apr 16 16:42:19.629295 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:42:19.629269    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c988f546f20bc9871eeda375c0f55e19d339c59f913c760c24f244eb3b673823\": container with ID starting with c988f546f20bc9871eeda375c0f55e19d339c59f913c760c24f244eb3b673823 not found: ID does not exist" containerID="c988f546f20bc9871eeda375c0f55e19d339c59f913c760c24f244eb3b673823"
Apr 16 16:42:19.629335 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:19.629304    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c988f546f20bc9871eeda375c0f55e19d339c59f913c760c24f244eb3b673823"} err="failed to get container status \"c988f546f20bc9871eeda375c0f55e19d339c59f913c760c24f244eb3b673823\": rpc error: code = NotFound desc = could not find container \"c988f546f20bc9871eeda375c0f55e19d339c59f913c760c24f244eb3b673823\": container with ID starting with c988f546f20bc9871eeda375c0f55e19d339c59f913c760c24f244eb3b673823 not found: ID does not exist"
Apr 16 16:42:21.235522 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:21.235492    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" path="/var/lib/kubelet/pods/a3148437-c0b1-44d3-9fea-3f7254ee765d/volumes"
Apr 16 16:42:21.557418 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:21.557341    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" podUID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Apr 16 16:42:31.557512 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:31.557473    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" podUID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Apr 16 16:42:41.557002 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:41.556959    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" podUID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Apr 16 16:42:51.557634 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:42:51.557594    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" podUID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Apr 16 16:43:01.556937 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:01.556851    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" podUID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Apr 16 16:43:03.231786 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:03.231743    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" podUID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Apr 16 16:43:13.236283 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:13.236252    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq"
Apr 16 16:43:18.988499 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:18.988466    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq"]
Apr 16 16:43:18.988963 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:18.988822    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" podUID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerName="kserve-container" containerID="cri-o://bb9d5961dabbf9977d7d99e16d5a7a53851e2e61dd953c4d7edd7282e9ba2ff9" gracePeriod=30
Apr 16 16:43:19.110596 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.110548    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm"]
Apr 16 16:43:19.110869 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.110857    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="kserve-container"
Apr 16 16:43:19.110913 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.110870    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="kserve-container"
Apr 16 16:43:19.110913 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.110880    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="storage-initializer"
Apr 16 16:43:19.110913 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.110886    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="storage-initializer"
Apr 16 16:43:19.110913 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.110905    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="agent"
Apr 16 16:43:19.110913 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.110911    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="agent"
Apr 16 16:43:19.111062 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.110960    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="agent"
Apr 16 16:43:19.111062 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.110969    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a3148437-c0b1-44d3-9fea-3f7254ee765d" containerName="kserve-container"
Apr 16 16:43:19.112924 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.112901    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm"
Apr 16 16:43:19.123081 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.123057    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm"]
Apr 16 16:43:19.195945 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.195916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8abd5bd-72af-4cbd-bd15-f9f4b926902d-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm\" (UID: \"b8abd5bd-72af-4cbd-bd15-f9f4b926902d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm"
Apr 16 16:43:19.296275 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.296211    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8abd5bd-72af-4cbd-bd15-f9f4b926902d-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm\" (UID: \"b8abd5bd-72af-4cbd-bd15-f9f4b926902d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm"
Apr 16 16:43:19.296531 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.296516    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8abd5bd-72af-4cbd-bd15-f9f4b926902d-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm\" (UID: \"b8abd5bd-72af-4cbd-bd15-f9f4b926902d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm"
Apr 16 16:43:19.422870 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.422845    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm"
Apr 16 16:43:19.536813 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.536783    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm"]
Apr 16 16:43:19.540144 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:43:19.540110    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8abd5bd_72af_4cbd_bd15_f9f4b926902d.slice/crio-9f1a05381d481cf7923ff6e3126c233e483b618746f1f2d9a60fab0c4e9baa22 WatchSource:0}: Error finding container 9f1a05381d481cf7923ff6e3126c233e483b618746f1f2d9a60fab0c4e9baa22: Status 404 returned error can't find the container with id 9f1a05381d481cf7923ff6e3126c233e483b618746f1f2d9a60fab0c4e9baa22
Apr 16 16:43:19.782424 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.782390    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm" event={"ID":"b8abd5bd-72af-4cbd-bd15-f9f4b926902d","Type":"ContainerStarted","Data":"d225880acb2ff23b69c20e0c0ccfb2eff2a6b20f7d565a832444829c4de4f966"}
Apr 16 16:43:19.782424 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:19.782429    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm" event={"ID":"b8abd5bd-72af-4cbd-bd15-f9f4b926902d","Type":"ContainerStarted","Data":"9f1a05381d481cf7923ff6e3126c233e483b618746f1f2d9a60fab0c4e9baa22"}
Apr 16 16:43:22.927851 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:22.927830    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq"
Apr 16 16:43:23.020925 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.020854    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de8da7ab-283e-409d-b9c3-e4a35ece25e3-kserve-provision-location\") pod \"de8da7ab-283e-409d-b9c3-e4a35ece25e3\" (UID: \"de8da7ab-283e-409d-b9c3-e4a35ece25e3\") "
Apr 16 16:43:23.021195 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.021171    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de8da7ab-283e-409d-b9c3-e4a35ece25e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de8da7ab-283e-409d-b9c3-e4a35ece25e3" (UID: "de8da7ab-283e-409d-b9c3-e4a35ece25e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:43:23.121895 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.121870    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de8da7ab-283e-409d-b9c3-e4a35ece25e3-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:43:23.796016 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.795983    2578 generic.go:358] "Generic (PLEG): container finished" podID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerID="bb9d5961dabbf9977d7d99e16d5a7a53851e2e61dd953c4d7edd7282e9ba2ff9" exitCode=0
Apr 16 16:43:23.796188 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.796075    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" event={"ID":"de8da7ab-283e-409d-b9c3-e4a35ece25e3","Type":"ContainerDied","Data":"bb9d5961dabbf9977d7d99e16d5a7a53851e2e61dd953c4d7edd7282e9ba2ff9"}
Apr 16 16:43:23.796188 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.796091    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq"
Apr 16 16:43:23.796188 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.796115    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq" event={"ID":"de8da7ab-283e-409d-b9c3-e4a35ece25e3","Type":"ContainerDied","Data":"cba18e347f107795e35d4e2d2dc1dbeb315033e99cb12ebba40a16a0fad2f939"}
Apr 16 16:43:23.796188 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.796135    2578 scope.go:117] "RemoveContainer" containerID="bb9d5961dabbf9977d7d99e16d5a7a53851e2e61dd953c4d7edd7282e9ba2ff9"
Apr 16 16:43:23.797571 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.797530    2578 generic.go:358] "Generic (PLEG): container finished" podID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" containerID="d225880acb2ff23b69c20e0c0ccfb2eff2a6b20f7d565a832444829c4de4f966" exitCode=0
Apr 16 16:43:23.797683 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.797590    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm" event={"ID":"b8abd5bd-72af-4cbd-bd15-f9f4b926902d","Type":"ContainerDied","Data":"d225880acb2ff23b69c20e0c0ccfb2eff2a6b20f7d565a832444829c4de4f966"}
Apr 16 16:43:23.804232 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.804049    2578 scope.go:117] "RemoveContainer" containerID="632ade3a10944edf1afe0b50b2286e76b1c86048f98f4665757fdda7676bc4bd"
Apr 16 16:43:23.810971 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.810953    2578 scope.go:117] "RemoveContainer" containerID="bb9d5961dabbf9977d7d99e16d5a7a53851e2e61dd953c4d7edd7282e9ba2ff9"
Apr 16 16:43:23.811219 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:43:23.811201    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9d5961dabbf9977d7d99e16d5a7a53851e2e61dd953c4d7edd7282e9ba2ff9\": container with ID starting with bb9d5961dabbf9977d7d99e16d5a7a53851e2e61dd953c4d7edd7282e9ba2ff9 not found: ID does not exist" containerID="bb9d5961dabbf9977d7d99e16d5a7a53851e2e61dd953c4d7edd7282e9ba2ff9"
Apr 16 16:43:23.811267 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.811227    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9d5961dabbf9977d7d99e16d5a7a53851e2e61dd953c4d7edd7282e9ba2ff9"} err="failed to get container status \"bb9d5961dabbf9977d7d99e16d5a7a53851e2e61dd953c4d7edd7282e9ba2ff9\": rpc error: code = NotFound desc = could not find container \"bb9d5961dabbf9977d7d99e16d5a7a53851e2e61dd953c4d7edd7282e9ba2ff9\": container with ID starting with bb9d5961dabbf9977d7d99e16d5a7a53851e2e61dd953c4d7edd7282e9ba2ff9 not found: ID does not exist"
Apr 16 16:43:23.811267 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.811244    2578 scope.go:117] "RemoveContainer" containerID="632ade3a10944edf1afe0b50b2286e76b1c86048f98f4665757fdda7676bc4bd"
Apr 16 16:43:23.811457 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:43:23.811444    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632ade3a10944edf1afe0b50b2286e76b1c86048f98f4665757fdda7676bc4bd\": container with ID starting with 632ade3a10944edf1afe0b50b2286e76b1c86048f98f4665757fdda7676bc4bd not found: ID does not exist" containerID="632ade3a10944edf1afe0b50b2286e76b1c86048f98f4665757fdda7676bc4bd"
Apr 16 16:43:23.811497 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.811460    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632ade3a10944edf1afe0b50b2286e76b1c86048f98f4665757fdda7676bc4bd"} err="failed to get container status \"632ade3a10944edf1afe0b50b2286e76b1c86048f98f4665757fdda7676bc4bd\": rpc error: code = NotFound desc = could not find container \"632ade3a10944edf1afe0b50b2286e76b1c86048f98f4665757fdda7676bc4bd\": container with ID starting with 632ade3a10944edf1afe0b50b2286e76b1c86048f98f4665757fdda7676bc4bd not found: ID does not exist"
Apr 16 16:43:23.841797 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.841769    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq"]
Apr 16 16:43:23.846084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:23.846065    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-dzzdq"]
Apr 16 16:43:24.803373 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:24.803343    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm" event={"ID":"b8abd5bd-72af-4cbd-bd15-f9f4b926902d","Type":"ContainerStarted","Data":"4f2b0875513569b803bdbe793a1a6cc6fe76eeaaffd53e2b06a62c4cf1b5ea05"}
Apr 16 16:43:24.803927 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:24.803663    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm"
Apr 16 16:43:24.804728 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:24.804703    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm" podUID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Apr 16 16:43:24.823863 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:24.823825    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm" podStartSLOduration=5.823813727 podStartE2EDuration="5.823813727s" podCreationTimestamp="2026-04-16 16:43:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:43:24.822738703 +0000 UTC m=+814.219955088" watchObservedRunningTime="2026-04-16 16:43:24.823813727 +0000 UTC m=+814.221030114"
Apr 16 16:43:25.236232 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:25.236203    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" path="/var/lib/kubelet/pods/de8da7ab-283e-409d-b9c3-e4a35ece25e3/volumes"
Apr 16 16:43:25.806989 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:25.806948    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm" podUID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Apr 16 16:43:35.807162 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:35.807119    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm" podUID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Apr 16 16:43:45.807616 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:45.807536    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm" podUID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Apr 16 16:43:55.807788 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:43:55.807746    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm" podUID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Apr 16 16:44:05.807607 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:05.807541    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm" podUID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Apr 16 16:44:15.807126 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:15.807080    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm" podUID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Apr 16 16:44:25.807545 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:25.807505    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm" podUID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Apr 16 16:44:35.808301 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:35.808217    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm"
Apr 16 16:44:40.409177 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:40.409146    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm"]
Apr 16 16:44:40.409647 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:40.409434    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm" podUID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" containerName="kserve-container" containerID="cri-o://4f2b0875513569b803bdbe793a1a6cc6fe76eeaaffd53e2b06a62c4cf1b5ea05" gracePeriod=30
Apr 16 16:44:40.467191 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:40.467164    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg"]
Apr 16 16:44:40.467472 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:40.467460    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerName="storage-initializer"
Apr 16 16:44:40.467572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:40.467474    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerName="storage-initializer"
Apr 16 16:44:40.467572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:40.467486    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerName="kserve-container"
Apr 16 16:44:40.467572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:40.467492    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerName="kserve-container"
Apr 16 16:44:40.467572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:40.467542    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="de8da7ab-283e-409d-b9c3-e4a35ece25e3" containerName="kserve-container"
Apr 16 16:44:40.470407 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:40.470392    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg"
Apr 16 16:44:40.471566 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:40.471532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32a68c1b-0850-4b5f-8d05-9b00995e6951-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg\" (UID: \"32a68c1b-0850-4b5f-8d05-9b00995e6951\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg"
Apr 16 16:44:40.482940 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:40.482920    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg"]
Apr 16 16:44:40.572666 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:40.572636    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32a68c1b-0850-4b5f-8d05-9b00995e6951-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg\" (UID: \"32a68c1b-0850-4b5f-8d05-9b00995e6951\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg"
Apr 16 16:44:40.572969 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:40.572954    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32a68c1b-0850-4b5f-8d05-9b00995e6951-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg\" (UID: \"32a68c1b-0850-4b5f-8d05-9b00995e6951\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg"
Apr 16 16:44:40.779892 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:40.779872    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg"
Apr 16 16:44:40.901946 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:40.901917    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg"]
Apr 16 16:44:40.904630 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:44:40.904594    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a68c1b_0850_4b5f_8d05_9b00995e6951.slice/crio-3e4608dd61205a234e538fb2cdc8a94f85bd563a1f09d4289f8f8fbf583acef7 WatchSource:0}: Error finding container 3e4608dd61205a234e538fb2cdc8a94f85bd563a1f09d4289f8f8fbf583acef7: Status 404 returned error can't find the container with id 3e4608dd61205a234e538fb2cdc8a94f85bd563a1f09d4289f8f8fbf583acef7
Apr 16 16:44:41.021816 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:41.021790    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg" event={"ID":"32a68c1b-0850-4b5f-8d05-9b00995e6951","Type":"ContainerStarted","Data":"b0d88775de8d472bdf88ed283415a15750e485dac9b773c8ba89165a6253aa34"}
Apr 16 16:44:41.021919 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:41.021821    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg" event={"ID":"32a68c1b-0850-4b5f-8d05-9b00995e6951","Type":"ContainerStarted","Data":"3e4608dd61205a234e538fb2cdc8a94f85bd563a1f09d4289f8f8fbf583acef7"}
Apr 16 16:44:44.239977 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:44.239957    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm"
Apr 16 16:44:44.300423 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:44.300372    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8abd5bd-72af-4cbd-bd15-f9f4b926902d-kserve-provision-location\") pod \"b8abd5bd-72af-4cbd-bd15-f9f4b926902d\" (UID: \"b8abd5bd-72af-4cbd-bd15-f9f4b926902d\") "
Apr 16 16:44:44.300703 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:44.300683    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8abd5bd-72af-4cbd-bd15-f9f4b926902d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8abd5bd-72af-4cbd-bd15-f9f4b926902d" (UID: "b8abd5bd-72af-4cbd-bd15-f9f4b926902d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:44:44.401510 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:44.401486    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8abd5bd-72af-4cbd-bd15-f9f4b926902d-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:44:45.033180 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:45.033148    2578 generic.go:358] "Generic (PLEG): container finished" podID="32a68c1b-0850-4b5f-8d05-9b00995e6951" containerID="b0d88775de8d472bdf88ed283415a15750e485dac9b773c8ba89165a6253aa34" exitCode=0
Apr 16 16:44:45.033302 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:45.033220    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg" event={"ID":"32a68c1b-0850-4b5f-8d05-9b00995e6951","Type":"ContainerDied","Data":"b0d88775de8d472bdf88ed283415a15750e485dac9b773c8ba89165a6253aa34"}
Apr 16 16:44:45.034688 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:45.034668    2578 generic.go:358] "Generic (PLEG): container finished" podID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" containerID="4f2b0875513569b803bdbe793a1a6cc6fe76eeaaffd53e2b06a62c4cf1b5ea05" exitCode=0
Apr 16 16:44:45.034762 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:45.034727    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm" event={"ID":"b8abd5bd-72af-4cbd-bd15-f9f4b926902d","Type":"ContainerDied","Data":"4f2b0875513569b803bdbe793a1a6cc6fe76eeaaffd53e2b06a62c4cf1b5ea05"}
Apr 16 16:44:45.034762 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:45.034742    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm"
Apr 16 16:44:45.034762 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:45.034756    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm" event={"ID":"b8abd5bd-72af-4cbd-bd15-f9f4b926902d","Type":"ContainerDied","Data":"9f1a05381d481cf7923ff6e3126c233e483b618746f1f2d9a60fab0c4e9baa22"}
Apr 16 16:44:45.034902 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:45.034774    2578 scope.go:117] "RemoveContainer" containerID="4f2b0875513569b803bdbe793a1a6cc6fe76eeaaffd53e2b06a62c4cf1b5ea05"
Apr 16 16:44:45.044479 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:45.044306    2578 scope.go:117] "RemoveContainer" containerID="d225880acb2ff23b69c20e0c0ccfb2eff2a6b20f7d565a832444829c4de4f966"
Apr 16 16:44:45.051305 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:45.051278    2578 scope.go:117] "RemoveContainer" containerID="4f2b0875513569b803bdbe793a1a6cc6fe76eeaaffd53e2b06a62c4cf1b5ea05"
Apr 16 16:44:45.051643 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:44:45.051615    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2b0875513569b803bdbe793a1a6cc6fe76eeaaffd53e2b06a62c4cf1b5ea05\": container with ID starting with 4f2b0875513569b803bdbe793a1a6cc6fe76eeaaffd53e2b06a62c4cf1b5ea05 not found: ID does not exist" containerID="4f2b0875513569b803bdbe793a1a6cc6fe76eeaaffd53e2b06a62c4cf1b5ea05"
Apr 16 16:44:45.051782 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:45.051670    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2b0875513569b803bdbe793a1a6cc6fe76eeaaffd53e2b06a62c4cf1b5ea05"} err="failed to get container status \"4f2b0875513569b803bdbe793a1a6cc6fe76eeaaffd53e2b06a62c4cf1b5ea05\": rpc error: code = NotFound desc = could not find container \"4f2b0875513569b803bdbe793a1a6cc6fe76eeaaffd53e2b06a62c4cf1b5ea05\": container with ID starting with 4f2b0875513569b803bdbe793a1a6cc6fe76eeaaffd53e2b06a62c4cf1b5ea05 not found: ID does not exist"
Apr 16 16:44:45.051782 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:45.051696    2578 scope.go:117] "RemoveContainer" containerID="d225880acb2ff23b69c20e0c0ccfb2eff2a6b20f7d565a832444829c4de4f966"
Apr 16 16:44:45.052118 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:44:45.052100    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d225880acb2ff23b69c20e0c0ccfb2eff2a6b20f7d565a832444829c4de4f966\": container with ID starting with d225880acb2ff23b69c20e0c0ccfb2eff2a6b20f7d565a832444829c4de4f966 not found: ID does not exist" containerID="d225880acb2ff23b69c20e0c0ccfb2eff2a6b20f7d565a832444829c4de4f966"
Apr 16 16:44:45.052174 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:45.052123    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d225880acb2ff23b69c20e0c0ccfb2eff2a6b20f7d565a832444829c4de4f966"} err="failed to get container status \"d225880acb2ff23b69c20e0c0ccfb2eff2a6b20f7d565a832444829c4de4f966\": rpc error: code = NotFound desc = could not find container \"d225880acb2ff23b69c20e0c0ccfb2eff2a6b20f7d565a832444829c4de4f966\": container with ID starting with d225880acb2ff23b69c20e0c0ccfb2eff2a6b20f7d565a832444829c4de4f966 not found: ID does not exist"
Apr 16 16:44:45.063672 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:45.063648    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm"]
Apr 16 16:44:45.069599 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:45.069580    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-wt5lm"]
Apr 16 16:44:45.234843 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:45.234816    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" path="/var/lib/kubelet/pods/b8abd5bd-72af-4cbd-bd15-f9f4b926902d/volumes"
Apr 16 16:44:51.173925 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:51.173894    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 16:44:51.175459 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:44:51.175378    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 16:46:54.449925 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:46:54.449890    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg" event={"ID":"32a68c1b-0850-4b5f-8d05-9b00995e6951","Type":"ContainerStarted","Data":"72bddd82d34eb656fcf7e546dcd48686954c21c399218ea8790a8b61e8633971"}
Apr 16 16:46:54.450411 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:46:54.450037    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg"
Apr 16 16:46:54.470476 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:46:54.470428    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg" podStartSLOduration=5.735172181 podStartE2EDuration="2m14.470416097s" podCreationTimestamp="2026-04-16 16:44:40 +0000 UTC" firstStartedPulling="2026-04-16 16:44:45.034229651 +0000 UTC m=+894.431446029" lastFinishedPulling="2026-04-16 16:46:53.769473565 +0000 UTC m=+1023.166689945" observedRunningTime="2026-04-16 16:46:54.469503563 +0000 UTC m=+1023.866719953" watchObservedRunningTime="2026-04-16 16:46:54.470416097 +0000 UTC m=+1023.867632485"
Apr 16 16:47:25.458111 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:25.458082    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg"
Apr 16 16:47:30.647873 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:30.647796    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg"]
Apr 16 16:47:30.648221 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:30.648055    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg" podUID="32a68c1b-0850-4b5f-8d05-9b00995e6951" containerName="kserve-container" containerID="cri-o://72bddd82d34eb656fcf7e546dcd48686954c21c399218ea8790a8b61e8633971" gracePeriod=30
Apr 16 16:47:30.727883 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:30.727839    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg"]
Apr 16 16:47:30.728245 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:30.728227    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" containerName="storage-initializer"
Apr 16 16:47:30.728330 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:30.728248    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" containerName="storage-initializer"
Apr 16 16:47:30.728330 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:30.728279    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" containerName="kserve-container"
Apr 16 16:47:30.728330 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:30.728288    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" containerName="kserve-container"
Apr 16 16:47:30.728478 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:30.728373    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="b8abd5bd-72af-4cbd-bd15-f9f4b926902d" containerName="kserve-container"
Apr 16 16:47:30.760530 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:30.760495    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg"]
Apr 16 16:47:30.760709 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:30.760655    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg"
Apr 16 16:47:30.930634 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:30.930564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088b9c08-3a81-4fed-bc7c-71acb928c8df-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg\" (UID: \"088b9c08-3a81-4fed-bc7c-71acb928c8df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg"
Apr 16 16:47:31.031931 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:31.031898    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088b9c08-3a81-4fed-bc7c-71acb928c8df-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg\" (UID: \"088b9c08-3a81-4fed-bc7c-71acb928c8df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg"
Apr 16 16:47:31.032214 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:31.032199    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088b9c08-3a81-4fed-bc7c-71acb928c8df-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg\" (UID: \"088b9c08-3a81-4fed-bc7c-71acb928c8df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg"
Apr 16 16:47:31.071114 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:31.071093    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg"
Apr 16 16:47:31.189291 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:31.189221    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg"]
Apr 16 16:47:31.192202 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:47:31.192171    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088b9c08_3a81_4fed_bc7c_71acb928c8df.slice/crio-1ceb6e21092855c784a23f341956bc551fa283bf8b5de34030e9c27f08d21ac0 WatchSource:0}: Error finding container 1ceb6e21092855c784a23f341956bc551fa283bf8b5de34030e9c27f08d21ac0: Status 404 returned error can't find the container with id 1ceb6e21092855c784a23f341956bc551fa283bf8b5de34030e9c27f08d21ac0
Apr 16 16:47:31.194217 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:31.194196    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 16:47:31.552242 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:31.552214    2578 generic.go:358] "Generic (PLEG): container finished" podID="32a68c1b-0850-4b5f-8d05-9b00995e6951" containerID="72bddd82d34eb656fcf7e546dcd48686954c21c399218ea8790a8b61e8633971" exitCode=0
Apr 16 16:47:31.552382 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:31.552288    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg" event={"ID":"32a68c1b-0850-4b5f-8d05-9b00995e6951","Type":"ContainerDied","Data":"72bddd82d34eb656fcf7e546dcd48686954c21c399218ea8790a8b61e8633971"}
Apr 16 16:47:31.553858 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:31.553836    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg" event={"ID":"088b9c08-3a81-4fed-bc7c-71acb928c8df","Type":"ContainerStarted","Data":"15112512fca2cfd7a152f8581dba88fd9989352195976bc6931c4cf6cd10330c"}
Apr 16 16:47:31.553982 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:31.553871    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg" event={"ID":"088b9c08-3a81-4fed-bc7c-71acb928c8df","Type":"ContainerStarted","Data":"1ceb6e21092855c784a23f341956bc551fa283bf8b5de34030e9c27f08d21ac0"}
Apr 16 16:47:31.594480 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:31.594456    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg"
Apr 16 16:47:31.738614 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:31.738587    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32a68c1b-0850-4b5f-8d05-9b00995e6951-kserve-provision-location\") pod \"32a68c1b-0850-4b5f-8d05-9b00995e6951\" (UID: \"32a68c1b-0850-4b5f-8d05-9b00995e6951\") "
Apr 16 16:47:31.738966 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:31.738867    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a68c1b-0850-4b5f-8d05-9b00995e6951-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32a68c1b-0850-4b5f-8d05-9b00995e6951" (UID: "32a68c1b-0850-4b5f-8d05-9b00995e6951"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:47:31.839958 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:31.839931    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32a68c1b-0850-4b5f-8d05-9b00995e6951-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:47:32.557895 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:32.557869    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg"
Apr 16 16:47:32.558023 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:32.557891    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg" event={"ID":"32a68c1b-0850-4b5f-8d05-9b00995e6951","Type":"ContainerDied","Data":"3e4608dd61205a234e538fb2cdc8a94f85bd563a1f09d4289f8f8fbf583acef7"}
Apr 16 16:47:32.558023 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:32.557942    2578 scope.go:117] "RemoveContainer" containerID="72bddd82d34eb656fcf7e546dcd48686954c21c399218ea8790a8b61e8633971"
Apr 16 16:47:32.566286 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:32.566265    2578 scope.go:117] "RemoveContainer" containerID="b0d88775de8d472bdf88ed283415a15750e485dac9b773c8ba89165a6253aa34"
Apr 16 16:47:32.578885 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:32.578864    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg"]
Apr 16 16:47:32.582407 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:32.582386    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xq5xg"]
Apr 16 16:47:33.235175 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:33.235149    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32a68c1b-0850-4b5f-8d05-9b00995e6951" path="/var/lib/kubelet/pods/32a68c1b-0850-4b5f-8d05-9b00995e6951/volumes"
Apr 16 16:47:35.568850 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:35.568819    2578 generic.go:358] "Generic (PLEG): container finished" podID="088b9c08-3a81-4fed-bc7c-71acb928c8df" containerID="15112512fca2cfd7a152f8581dba88fd9989352195976bc6931c4cf6cd10330c" exitCode=0
Apr 16 16:47:35.569156 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:35.568889    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg" event={"ID":"088b9c08-3a81-4fed-bc7c-71acb928c8df","Type":"ContainerDied","Data":"15112512fca2cfd7a152f8581dba88fd9989352195976bc6931c4cf6cd10330c"}
Apr 16 16:47:36.573521 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:36.573485    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg" event={"ID":"088b9c08-3a81-4fed-bc7c-71acb928c8df","Type":"ContainerStarted","Data":"9bd7f2481ac64a9333891da3cb9937e42d5290056508eca591214855e8060a5b"}
Apr 16 16:47:36.573906 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:36.573793    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg"
Apr 16 16:47:36.575106 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:36.575079    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg" podUID="088b9c08-3a81-4fed-bc7c-71acb928c8df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Apr 16 16:47:36.589156 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:36.589118    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg" podStartSLOduration=6.589108143 podStartE2EDuration="6.589108143s" podCreationTimestamp="2026-04-16 16:47:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:47:36.588011808 +0000 UTC m=+1065.985228195" watchObservedRunningTime="2026-04-16 16:47:36.589108143 +0000 UTC m=+1065.986324531"
Apr 16 16:47:37.577258 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:37.577218    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg" podUID="088b9c08-3a81-4fed-bc7c-71acb928c8df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Apr 16 16:47:47.578756 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:47.578719    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg"
Apr 16 16:47:50.818819 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:50.818780    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg"]
Apr 16 16:47:50.819212 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:50.819072    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg" podUID="088b9c08-3a81-4fed-bc7c-71acb928c8df" containerName="kserve-container" containerID="cri-o://9bd7f2481ac64a9333891da3cb9937e42d5290056508eca591214855e8060a5b" gracePeriod=30
Apr 16 16:47:50.888345 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:50.888314    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8"]
Apr 16 16:47:50.888636 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:50.888623    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a68c1b-0850-4b5f-8d05-9b00995e6951" containerName="kserve-container"
Apr 16 16:47:50.888699 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:50.888637    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a68c1b-0850-4b5f-8d05-9b00995e6951" containerName="kserve-container"
Apr 16 16:47:50.888699 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:50.888653    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a68c1b-0850-4b5f-8d05-9b00995e6951" containerName="storage-initializer"
Apr 16 16:47:50.888699 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:50.888660    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a68c1b-0850-4b5f-8d05-9b00995e6951" containerName="storage-initializer"
Apr 16 16:47:50.888802 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:50.888706    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="32a68c1b-0850-4b5f-8d05-9b00995e6951" containerName="kserve-container"
Apr 16 16:47:50.893928 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:50.893900    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8"
Apr 16 16:47:50.901755 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:50.901729    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8"]
Apr 16 16:47:51.076610 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.076502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1097327d-d990-4491-933d-ec5a8545f086-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8\" (UID: \"1097327d-d990-4491-933d-ec5a8545f086\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8"
Apr 16 16:47:51.177783 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.177749    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1097327d-d990-4491-933d-ec5a8545f086-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8\" (UID: \"1097327d-d990-4491-933d-ec5a8545f086\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8"
Apr 16 16:47:51.178105 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.178085    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1097327d-d990-4491-933d-ec5a8545f086-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8\" (UID: \"1097327d-d990-4491-933d-ec5a8545f086\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8"
Apr 16 16:47:51.205592 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.205548    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8"
Apr 16 16:47:51.328966 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.328906    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8"]
Apr 16 16:47:51.331699 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:47:51.331668    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1097327d_d990_4491_933d_ec5a8545f086.slice/crio-6a2a088212ce5c58fe93804b7c1b6ee29e6311657d505fca74830f2cfe31f2b3 WatchSource:0}: Error finding container 6a2a088212ce5c58fe93804b7c1b6ee29e6311657d505fca74830f2cfe31f2b3: Status 404 returned error can't find the container with id 6a2a088212ce5c58fe93804b7c1b6ee29e6311657d505fca74830f2cfe31f2b3
Apr 16 16:47:51.446187 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.446162    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg"
Apr 16 16:47:51.581078 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.581017    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088b9c08-3a81-4fed-bc7c-71acb928c8df-kserve-provision-location\") pod \"088b9c08-3a81-4fed-bc7c-71acb928c8df\" (UID: \"088b9c08-3a81-4fed-bc7c-71acb928c8df\") "
Apr 16 16:47:51.581336 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.581317    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088b9c08-3a81-4fed-bc7c-71acb928c8df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "088b9c08-3a81-4fed-bc7c-71acb928c8df" (UID: "088b9c08-3a81-4fed-bc7c-71acb928c8df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:47:51.616889 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.616853    2578 generic.go:358] "Generic (PLEG): container finished" podID="088b9c08-3a81-4fed-bc7c-71acb928c8df" containerID="9bd7f2481ac64a9333891da3cb9937e42d5290056508eca591214855e8060a5b" exitCode=0
Apr 16 16:47:51.617017 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.616924    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg"
Apr 16 16:47:51.617070 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.616913    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg" event={"ID":"088b9c08-3a81-4fed-bc7c-71acb928c8df","Type":"ContainerDied","Data":"9bd7f2481ac64a9333891da3cb9937e42d5290056508eca591214855e8060a5b"}
Apr 16 16:47:51.617070 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.617058    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg" event={"ID":"088b9c08-3a81-4fed-bc7c-71acb928c8df","Type":"ContainerDied","Data":"1ceb6e21092855c784a23f341956bc551fa283bf8b5de34030e9c27f08d21ac0"}
Apr 16 16:47:51.617170 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.617086    2578 scope.go:117] "RemoveContainer" containerID="9bd7f2481ac64a9333891da3cb9937e42d5290056508eca591214855e8060a5b"
Apr 16 16:47:51.618490 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.618468    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8" event={"ID":"1097327d-d990-4491-933d-ec5a8545f086","Type":"ContainerStarted","Data":"85e429361df1d2cd68d3e5c2a2b13f5dc9f268d492821bffb83b53306d68854e"}
Apr 16 16:47:51.618618 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.618498    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8" event={"ID":"1097327d-d990-4491-933d-ec5a8545f086","Type":"ContainerStarted","Data":"6a2a088212ce5c58fe93804b7c1b6ee29e6311657d505fca74830f2cfe31f2b3"}
Apr 16 16:47:51.625201 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.625184    2578 scope.go:117] "RemoveContainer" containerID="15112512fca2cfd7a152f8581dba88fd9989352195976bc6931c4cf6cd10330c"
Apr 16 16:47:51.633497 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.633479    2578 scope.go:117] "RemoveContainer" containerID="9bd7f2481ac64a9333891da3cb9937e42d5290056508eca591214855e8060a5b"
Apr 16 16:47:51.633773 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:47:51.633748    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd7f2481ac64a9333891da3cb9937e42d5290056508eca591214855e8060a5b\": container with ID starting with 9bd7f2481ac64a9333891da3cb9937e42d5290056508eca591214855e8060a5b not found: ID does not exist" containerID="9bd7f2481ac64a9333891da3cb9937e42d5290056508eca591214855e8060a5b"
Apr 16 16:47:51.633852 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.633781    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd7f2481ac64a9333891da3cb9937e42d5290056508eca591214855e8060a5b"} err="failed to get container status \"9bd7f2481ac64a9333891da3cb9937e42d5290056508eca591214855e8060a5b\": rpc error: code = NotFound desc = could not find container \"9bd7f2481ac64a9333891da3cb9937e42d5290056508eca591214855e8060a5b\": container with ID starting with 9bd7f2481ac64a9333891da3cb9937e42d5290056508eca591214855e8060a5b not found: ID does not exist"
Apr 16 16:47:51.633852 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.633797    2578 scope.go:117] "RemoveContainer" containerID="15112512fca2cfd7a152f8581dba88fd9989352195976bc6931c4cf6cd10330c"
Apr 16 16:47:51.634092 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:47:51.634036    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15112512fca2cfd7a152f8581dba88fd9989352195976bc6931c4cf6cd10330c\": container with ID starting with 15112512fca2cfd7a152f8581dba88fd9989352195976bc6931c4cf6cd10330c not found: ID does not exist" containerID="15112512fca2cfd7a152f8581dba88fd9989352195976bc6931c4cf6cd10330c"
Apr 16 16:47:51.634092 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.634070    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15112512fca2cfd7a152f8581dba88fd9989352195976bc6931c4cf6cd10330c"} err="failed to get container status \"15112512fca2cfd7a152f8581dba88fd9989352195976bc6931c4cf6cd10330c\": rpc error: code = NotFound desc = could not find container \"15112512fca2cfd7a152f8581dba88fd9989352195976bc6931c4cf6cd10330c\": container with ID starting with 15112512fca2cfd7a152f8581dba88fd9989352195976bc6931c4cf6cd10330c not found: ID does not exist"
Apr 16 16:47:51.644715 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.644695    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg"]
Apr 16 16:47:51.648221 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.648205    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-nd4hg"]
Apr 16 16:47:51.681847 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:51.681823    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088b9c08-3a81-4fed-bc7c-71acb928c8df-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:47:53.234693 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:53.234658    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="088b9c08-3a81-4fed-bc7c-71acb928c8df" path="/var/lib/kubelet/pods/088b9c08-3a81-4fed-bc7c-71acb928c8df/volumes"
Apr 16 16:47:55.631359 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:55.631291    2578 generic.go:358] "Generic (PLEG): container finished" podID="1097327d-d990-4491-933d-ec5a8545f086" containerID="85e429361df1d2cd68d3e5c2a2b13f5dc9f268d492821bffb83b53306d68854e" exitCode=0
Apr 16 16:47:55.631359 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:55.631331    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8" event={"ID":"1097327d-d990-4491-933d-ec5a8545f086","Type":"ContainerDied","Data":"85e429361df1d2cd68d3e5c2a2b13f5dc9f268d492821bffb83b53306d68854e"}
Apr 16 16:47:56.635915 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:56.635870    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8" event={"ID":"1097327d-d990-4491-933d-ec5a8545f086","Type":"ContainerStarted","Data":"aee172773d20a363caf2ad9d69adc264710c528681a3172b274c733a1887769d"}
Apr 16 16:47:56.636425 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:56.636095    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8"
Apr 16 16:47:56.654806 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:47:56.654764    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8" podStartSLOduration=6.65475263 podStartE2EDuration="6.65475263s" podCreationTimestamp="2026-04-16 16:47:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:47:56.653362085 +0000 UTC m=+1086.050578473" watchObservedRunningTime="2026-04-16 16:47:56.65475263 +0000 UTC m=+1086.051969019"
Apr 16 16:48:27.643628 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:27.643600    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8"
Apr 16 16:48:30.989426 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:30.989393    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8"]
Apr 16 16:48:30.989831 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:30.989768    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8" podUID="1097327d-d990-4491-933d-ec5a8545f086" containerName="kserve-container" containerID="cri-o://aee172773d20a363caf2ad9d69adc264710c528681a3172b274c733a1887769d" gracePeriod=30
Apr 16 16:48:31.062803 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:31.062774    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"]
Apr 16 16:48:31.063120 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:31.063106    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="088b9c08-3a81-4fed-bc7c-71acb928c8df" containerName="storage-initializer"
Apr 16 16:48:31.063165 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:31.063122    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="088b9c08-3a81-4fed-bc7c-71acb928c8df" containerName="storage-initializer"
Apr 16 16:48:31.063165 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:31.063140    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="088b9c08-3a81-4fed-bc7c-71acb928c8df" containerName="kserve-container"
Apr 16 16:48:31.063165 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:31.063146    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="088b9c08-3a81-4fed-bc7c-71acb928c8df" containerName="kserve-container"
Apr 16 16:48:31.063255 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:31.063198    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="088b9c08-3a81-4fed-bc7c-71acb928c8df" containerName="kserve-container"
Apr 16 16:48:31.066426 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:31.066411    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"
Apr 16 16:48:31.088329 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:31.088302    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"]
Apr 16 16:48:31.154145 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:31.154116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bea9688-1f70-49bd-b046-e68c69ffd3bf-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h\" (UID: \"5bea9688-1f70-49bd-b046-e68c69ffd3bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"
Apr 16 16:48:31.255342 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:31.255263    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bea9688-1f70-49bd-b046-e68c69ffd3bf-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h\" (UID: \"5bea9688-1f70-49bd-b046-e68c69ffd3bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"
Apr 16 16:48:31.255633 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:31.255616    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bea9688-1f70-49bd-b046-e68c69ffd3bf-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h\" (UID: \"5bea9688-1f70-49bd-b046-e68c69ffd3bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"
Apr 16 16:48:31.375721 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:31.375674    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"
Apr 16 16:48:31.494923 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:31.494897    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"]
Apr 16 16:48:31.498305 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:48:31.498279    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bea9688_1f70_49bd_b046_e68c69ffd3bf.slice/crio-909f90ac63870864629fa0a89ce14c886075b4420429a859ec407630bd2269af WatchSource:0}: Error finding container 909f90ac63870864629fa0a89ce14c886075b4420429a859ec407630bd2269af: Status 404 returned error can't find the container with id 909f90ac63870864629fa0a89ce14c886075b4420429a859ec407630bd2269af
Apr 16 16:48:31.734195 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:31.734156    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h" event={"ID":"5bea9688-1f70-49bd-b046-e68c69ffd3bf","Type":"ContainerStarted","Data":"20646028ed0014b3d31919ff814cc3568e2274f441443e5de1e7c1f20c3e1ed7"}
Apr 16 16:48:31.734419 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:31.734204    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h" event={"ID":"5bea9688-1f70-49bd-b046-e68c69ffd3bf","Type":"ContainerStarted","Data":"909f90ac63870864629fa0a89ce14c886075b4420429a859ec407630bd2269af"}
Apr 16 16:48:32.116238 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:32.116180    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8"
Apr 16 16:48:32.163951 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:32.163924    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1097327d-d990-4491-933d-ec5a8545f086-kserve-provision-location\") pod \"1097327d-d990-4491-933d-ec5a8545f086\" (UID: \"1097327d-d990-4491-933d-ec5a8545f086\") "
Apr 16 16:48:32.164235 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:32.164214    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1097327d-d990-4491-933d-ec5a8545f086-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1097327d-d990-4491-933d-ec5a8545f086" (UID: "1097327d-d990-4491-933d-ec5a8545f086"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:48:32.264600 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:32.264578    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1097327d-d990-4491-933d-ec5a8545f086-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:48:32.738636 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:32.738603    2578 generic.go:358] "Generic (PLEG): container finished" podID="1097327d-d990-4491-933d-ec5a8545f086" containerID="aee172773d20a363caf2ad9d69adc264710c528681a3172b274c733a1887769d" exitCode=0
Apr 16 16:48:32.738777 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:32.738676    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8"
Apr 16 16:48:32.738777 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:32.738680    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8" event={"ID":"1097327d-d990-4491-933d-ec5a8545f086","Type":"ContainerDied","Data":"aee172773d20a363caf2ad9d69adc264710c528681a3172b274c733a1887769d"}
Apr 16 16:48:32.738777 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:32.738718    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8" event={"ID":"1097327d-d990-4491-933d-ec5a8545f086","Type":"ContainerDied","Data":"6a2a088212ce5c58fe93804b7c1b6ee29e6311657d505fca74830f2cfe31f2b3"}
Apr 16 16:48:32.738777 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:32.738732    2578 scope.go:117] "RemoveContainer" containerID="aee172773d20a363caf2ad9d69adc264710c528681a3172b274c733a1887769d"
Apr 16 16:48:32.747081 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:32.747066    2578 scope.go:117] "RemoveContainer" containerID="85e429361df1d2cd68d3e5c2a2b13f5dc9f268d492821bffb83b53306d68854e"
Apr 16 16:48:32.753860 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:32.753844    2578 scope.go:117] "RemoveContainer" containerID="aee172773d20a363caf2ad9d69adc264710c528681a3172b274c733a1887769d"
Apr 16 16:48:32.754098 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:48:32.754076    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee172773d20a363caf2ad9d69adc264710c528681a3172b274c733a1887769d\": container with ID starting with aee172773d20a363caf2ad9d69adc264710c528681a3172b274c733a1887769d not found: ID does not exist" containerID="aee172773d20a363caf2ad9d69adc264710c528681a3172b274c733a1887769d"
Apr 16 16:48:32.754181 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:32.754108    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee172773d20a363caf2ad9d69adc264710c528681a3172b274c733a1887769d"} err="failed to get container status \"aee172773d20a363caf2ad9d69adc264710c528681a3172b274c733a1887769d\": rpc error: code = NotFound desc = could not find container \"aee172773d20a363caf2ad9d69adc264710c528681a3172b274c733a1887769d\": container with ID starting with aee172773d20a363caf2ad9d69adc264710c528681a3172b274c733a1887769d not found: ID does not exist"
Apr 16 16:48:32.754181 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:32.754131    2578 scope.go:117] "RemoveContainer" containerID="85e429361df1d2cd68d3e5c2a2b13f5dc9f268d492821bffb83b53306d68854e"
Apr 16 16:48:32.754345 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:48:32.754330    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e429361df1d2cd68d3e5c2a2b13f5dc9f268d492821bffb83b53306d68854e\": container with ID starting with 85e429361df1d2cd68d3e5c2a2b13f5dc9f268d492821bffb83b53306d68854e not found: ID does not exist" containerID="85e429361df1d2cd68d3e5c2a2b13f5dc9f268d492821bffb83b53306d68854e"
Apr 16 16:48:32.754387 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:32.754350    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e429361df1d2cd68d3e5c2a2b13f5dc9f268d492821bffb83b53306d68854e"} err="failed to get container status \"85e429361df1d2cd68d3e5c2a2b13f5dc9f268d492821bffb83b53306d68854e\": rpc error: code = NotFound desc = could not find container \"85e429361df1d2cd68d3e5c2a2b13f5dc9f268d492821bffb83b53306d68854e\": container with ID starting with 85e429361df1d2cd68d3e5c2a2b13f5dc9f268d492821bffb83b53306d68854e not found: ID does not exist"
Apr 16 16:48:32.758167 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:32.758146    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8"]
Apr 16 16:48:32.761645 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:32.761626    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-27qt8"]
Apr 16 16:48:33.234931 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:33.234904    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1097327d-d990-4491-933d-ec5a8545f086" path="/var/lib/kubelet/pods/1097327d-d990-4491-933d-ec5a8545f086/volumes"
Apr 16 16:48:35.748396 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:35.748338    2578 generic.go:358] "Generic (PLEG): container finished" podID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" containerID="20646028ed0014b3d31919ff814cc3568e2274f441443e5de1e7c1f20c3e1ed7" exitCode=0
Apr 16 16:48:35.748713 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:35.748410    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h" event={"ID":"5bea9688-1f70-49bd-b046-e68c69ffd3bf","Type":"ContainerDied","Data":"20646028ed0014b3d31919ff814cc3568e2274f441443e5de1e7c1f20c3e1ed7"}
Apr 16 16:48:36.752996 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:36.752967    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h" event={"ID":"5bea9688-1f70-49bd-b046-e68c69ffd3bf","Type":"ContainerStarted","Data":"8bb8435c012fc3746520c9892ceba004ffae49423b26bdc014fa6151f49dfb61"}
Apr 16 16:48:38.761167 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:38.761131    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h" event={"ID":"5bea9688-1f70-49bd-b046-e68c69ffd3bf","Type":"ContainerStarted","Data":"3311801af70185e4ad5e98a152e65c337cba616f954b086c6f2d6891d071a140"}
Apr 16 16:48:38.761645 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:38.761292    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"
Apr 16 16:48:38.778047 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:38.777998    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h" podStartSLOduration=5.441650887 podStartE2EDuration="7.777981866s" podCreationTimestamp="2026-04-16 16:48:31 +0000 UTC" firstStartedPulling="2026-04-16 16:48:35.807115642 +0000 UTC m=+1125.204332008" lastFinishedPulling="2026-04-16 16:48:38.143446618 +0000 UTC m=+1127.540662987" observedRunningTime="2026-04-16 16:48:38.776471882 +0000 UTC m=+1128.173688284" watchObservedRunningTime="2026-04-16 16:48:38.777981866 +0000 UTC m=+1128.175198255"
Apr 16 16:48:39.764722 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:48:39.764693    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"
Apr 16 16:49:10.769869 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:10.769838    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"
Apr 16 16:49:40.771328 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:40.771300    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"
Apr 16 16:49:41.191787 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.191721    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"]
Apr 16 16:49:41.191990 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.191970    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h" podUID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" containerName="kserve-container" containerID="cri-o://8bb8435c012fc3746520c9892ceba004ffae49423b26bdc014fa6151f49dfb61" gracePeriod=30
Apr 16 16:49:41.192101 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.192058    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h" podUID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" containerName="kserve-agent" containerID="cri-o://3311801af70185e4ad5e98a152e65c337cba616f954b086c6f2d6891d071a140" gracePeriod=30
Apr 16 16:49:41.264697 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.264669    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf"]
Apr 16 16:49:41.265002 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.264988    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1097327d-d990-4491-933d-ec5a8545f086" containerName="kserve-container"
Apr 16 16:49:41.265049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.265003    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1097327d-d990-4491-933d-ec5a8545f086" containerName="kserve-container"
Apr 16 16:49:41.265049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.265023    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1097327d-d990-4491-933d-ec5a8545f086" containerName="storage-initializer"
Apr 16 16:49:41.265049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.265029    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1097327d-d990-4491-933d-ec5a8545f086" containerName="storage-initializer"
Apr 16 16:49:41.265138 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.265082    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="1097327d-d990-4491-933d-ec5a8545f086" containerName="kserve-container"
Apr 16 16:49:41.267501 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.267486    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf"
Apr 16 16:49:41.277383 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.277363    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf"]
Apr 16 16:49:41.358307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.358281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d711f7c-da57-4df7-bbf0-19670ac3b317-kserve-provision-location\") pod \"isvc-paddle-predictor-7dddcb4bd4-288rf\" (UID: \"3d711f7c-da57-4df7-bbf0-19670ac3b317\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf"
Apr 16 16:49:41.459363 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.459336    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d711f7c-da57-4df7-bbf0-19670ac3b317-kserve-provision-location\") pod \"isvc-paddle-predictor-7dddcb4bd4-288rf\" (UID: \"3d711f7c-da57-4df7-bbf0-19670ac3b317\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf"
Apr 16 16:49:41.459669 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.459653    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d711f7c-da57-4df7-bbf0-19670ac3b317-kserve-provision-location\") pod \"isvc-paddle-predictor-7dddcb4bd4-288rf\" (UID: \"3d711f7c-da57-4df7-bbf0-19670ac3b317\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf"
Apr 16 16:49:41.577693 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.577662    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf"
Apr 16 16:49:41.693784 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.693762    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf"]
Apr 16 16:49:41.696150 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:49:41.696121    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d711f7c_da57_4df7_bbf0_19670ac3b317.slice/crio-93c18d032b90e419972118dc1356f2d1b48d62ac70366ee0a91a5b41e98fde69 WatchSource:0}: Error finding container 93c18d032b90e419972118dc1356f2d1b48d62ac70366ee0a91a5b41e98fde69: Status 404 returned error can't find the container with id 93c18d032b90e419972118dc1356f2d1b48d62ac70366ee0a91a5b41e98fde69
Apr 16 16:49:41.941334 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.941298    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf" event={"ID":"3d711f7c-da57-4df7-bbf0-19670ac3b317","Type":"ContainerStarted","Data":"0eba1d9aff6fb8d35ef91804676a4175ff77916a476d59f08fdad43b68fbe6ba"}
Apr 16 16:49:41.941334 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:41.941338    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf" event={"ID":"3d711f7c-da57-4df7-bbf0-19670ac3b317","Type":"ContainerStarted","Data":"93c18d032b90e419972118dc1356f2d1b48d62ac70366ee0a91a5b41e98fde69"}
Apr 16 16:49:43.949186 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:43.949154    2578 generic.go:358] "Generic (PLEG): container finished" podID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" containerID="8bb8435c012fc3746520c9892ceba004ffae49423b26bdc014fa6151f49dfb61" exitCode=0
Apr 16 16:49:43.949574 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:43.949224    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h" event={"ID":"5bea9688-1f70-49bd-b046-e68c69ffd3bf","Type":"ContainerDied","Data":"8bb8435c012fc3746520c9892ceba004ffae49423b26bdc014fa6151f49dfb61"}
Apr 16 16:49:46.960158 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:46.960123    2578 generic.go:358] "Generic (PLEG): container finished" podID="3d711f7c-da57-4df7-bbf0-19670ac3b317" containerID="0eba1d9aff6fb8d35ef91804676a4175ff77916a476d59f08fdad43b68fbe6ba" exitCode=0
Apr 16 16:49:46.960577 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:46.960195    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf" event={"ID":"3d711f7c-da57-4df7-bbf0-19670ac3b317","Type":"ContainerDied","Data":"0eba1d9aff6fb8d35ef91804676a4175ff77916a476d59f08fdad43b68fbe6ba"}
Apr 16 16:49:50.767400 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:50.767349    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h" podUID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.35:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.35:8080: connect: connection refused"
Apr 16 16:49:51.202923 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:51.202894    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 16:49:51.204654 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:51.204633    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 16:49:58.999051 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:58.999020    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf" event={"ID":"3d711f7c-da57-4df7-bbf0-19670ac3b317","Type":"ContainerStarted","Data":"9812db8729bdd0fd25bd826765e44573c337afeca6dc69d67e4e5f57037fd78e"}
Apr 16 16:49:58.999374 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:58.999286    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf"
Apr 16 16:49:59.000518 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:59.000491    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf" podUID="3d711f7c-da57-4df7-bbf0-19670ac3b317" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Apr 16 16:49:59.015647 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:49:59.015608    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf" podStartSLOduration=6.349565646 podStartE2EDuration="18.015595254s" podCreationTimestamp="2026-04-16 16:49:41 +0000 UTC" firstStartedPulling="2026-04-16 16:49:46.961435015 +0000 UTC m=+1196.358651380" lastFinishedPulling="2026-04-16 16:49:58.627464618 +0000 UTC m=+1208.024680988" observedRunningTime="2026-04-16 16:49:59.013678778 +0000 UTC m=+1208.410895167" watchObservedRunningTime="2026-04-16 16:49:59.015595254 +0000 UTC m=+1208.412811642"
Apr 16 16:50:00.002645 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:00.002604    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf" podUID="3d711f7c-da57-4df7-bbf0-19670ac3b317" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Apr 16 16:50:00.768169 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:00.768132    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h" podUID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.35:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.35:8080: connect: connection refused"
Apr 16 16:50:10.002891 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:10.002848    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf" podUID="3d711f7c-da57-4df7-bbf0-19670ac3b317" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Apr 16 16:50:10.768275 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:10.768237    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h" podUID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.35:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.35:8080: connect: connection refused"
Apr 16 16:50:10.768436 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:10.768356    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"
Apr 16 16:50:11.373665 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:11.373641    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"
Apr 16 16:50:11.494792 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:11.494770    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bea9688-1f70-49bd-b046-e68c69ffd3bf-kserve-provision-location\") pod \"5bea9688-1f70-49bd-b046-e68c69ffd3bf\" (UID: \"5bea9688-1f70-49bd-b046-e68c69ffd3bf\") "
Apr 16 16:50:11.495065 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:11.495043    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bea9688-1f70-49bd-b046-e68c69ffd3bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5bea9688-1f70-49bd-b046-e68c69ffd3bf" (UID: "5bea9688-1f70-49bd-b046-e68c69ffd3bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:50:11.596246 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:11.596226    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bea9688-1f70-49bd-b046-e68c69ffd3bf-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:50:12.034536 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:12.034504    2578 generic.go:358] "Generic (PLEG): container finished" podID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" containerID="3311801af70185e4ad5e98a152e65c337cba616f954b086c6f2d6891d071a140" exitCode=0
Apr 16 16:50:12.034725 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:12.034571    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h" event={"ID":"5bea9688-1f70-49bd-b046-e68c69ffd3bf","Type":"ContainerDied","Data":"3311801af70185e4ad5e98a152e65c337cba616f954b086c6f2d6891d071a140"}
Apr 16 16:50:12.034725 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:12.034602    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h" event={"ID":"5bea9688-1f70-49bd-b046-e68c69ffd3bf","Type":"ContainerDied","Data":"909f90ac63870864629fa0a89ce14c886075b4420429a859ec407630bd2269af"}
Apr 16 16:50:12.034725 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:12.034602    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"
Apr 16 16:50:12.034725 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:12.034617    2578 scope.go:117] "RemoveContainer" containerID="3311801af70185e4ad5e98a152e65c337cba616f954b086c6f2d6891d071a140"
Apr 16 16:50:12.043445 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:12.043428    2578 scope.go:117] "RemoveContainer" containerID="8bb8435c012fc3746520c9892ceba004ffae49423b26bdc014fa6151f49dfb61"
Apr 16 16:50:12.050080 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:12.050060    2578 scope.go:117] "RemoveContainer" containerID="20646028ed0014b3d31919ff814cc3568e2274f441443e5de1e7c1f20c3e1ed7"
Apr 16 16:50:12.055475 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:12.055454    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"]
Apr 16 16:50:12.056913 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:12.056898    2578 scope.go:117] "RemoveContainer" containerID="3311801af70185e4ad5e98a152e65c337cba616f954b086c6f2d6891d071a140"
Apr 16 16:50:12.057122 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:50:12.057106    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3311801af70185e4ad5e98a152e65c337cba616f954b086c6f2d6891d071a140\": container with ID starting with 3311801af70185e4ad5e98a152e65c337cba616f954b086c6f2d6891d071a140 not found: ID does not exist" containerID="3311801af70185e4ad5e98a152e65c337cba616f954b086c6f2d6891d071a140"
Apr 16 16:50:12.057169 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:12.057131    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3311801af70185e4ad5e98a152e65c337cba616f954b086c6f2d6891d071a140"} err="failed to get container status \"3311801af70185e4ad5e98a152e65c337cba616f954b086c6f2d6891d071a140\": rpc error: code = NotFound desc = could not find container \"3311801af70185e4ad5e98a152e65c337cba616f954b086c6f2d6891d071a140\": container with ID starting with 3311801af70185e4ad5e98a152e65c337cba616f954b086c6f2d6891d071a140 not found: ID does not exist"
Apr 16 16:50:12.057169 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:12.057148    2578 scope.go:117] "RemoveContainer" containerID="8bb8435c012fc3746520c9892ceba004ffae49423b26bdc014fa6151f49dfb61"
Apr 16 16:50:12.057385 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:50:12.057369    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb8435c012fc3746520c9892ceba004ffae49423b26bdc014fa6151f49dfb61\": container with ID starting with 8bb8435c012fc3746520c9892ceba004ffae49423b26bdc014fa6151f49dfb61 not found: ID does not exist" containerID="8bb8435c012fc3746520c9892ceba004ffae49423b26bdc014fa6151f49dfb61"
Apr 16 16:50:12.057429 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:12.057390    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb8435c012fc3746520c9892ceba004ffae49423b26bdc014fa6151f49dfb61"} err="failed to get container status \"8bb8435c012fc3746520c9892ceba004ffae49423b26bdc014fa6151f49dfb61\": rpc error: code = NotFound desc = could not find container \"8bb8435c012fc3746520c9892ceba004ffae49423b26bdc014fa6151f49dfb61\": container with ID starting with 8bb8435c012fc3746520c9892ceba004ffae49423b26bdc014fa6151f49dfb61 not found: ID does not exist"
Apr 16 16:50:12.057429 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:12.057404    2578 scope.go:117] "RemoveContainer" containerID="20646028ed0014b3d31919ff814cc3568e2274f441443e5de1e7c1f20c3e1ed7"
Apr 16 16:50:12.057618 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:50:12.057602    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20646028ed0014b3d31919ff814cc3568e2274f441443e5de1e7c1f20c3e1ed7\": container with ID starting with 20646028ed0014b3d31919ff814cc3568e2274f441443e5de1e7c1f20c3e1ed7 not found: ID does not exist" containerID="20646028ed0014b3d31919ff814cc3568e2274f441443e5de1e7c1f20c3e1ed7"
Apr 16 16:50:12.057690 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:12.057625    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20646028ed0014b3d31919ff814cc3568e2274f441443e5de1e7c1f20c3e1ed7"} err="failed to get container status \"20646028ed0014b3d31919ff814cc3568e2274f441443e5de1e7c1f20c3e1ed7\": rpc error: code = NotFound desc = could not find container \"20646028ed0014b3d31919ff814cc3568e2274f441443e5de1e7c1f20c3e1ed7\": container with ID starting with 20646028ed0014b3d31919ff814cc3568e2274f441443e5de1e7c1f20c3e1ed7 not found: ID does not exist"
Apr 16 16:50:12.059810 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:12.059791    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bb8d86b55-k4x5h"]
Apr 16 16:50:13.234879 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:13.234844    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" path="/var/lib/kubelet/pods/5bea9688-1f70-49bd-b046-e68c69ffd3bf/volumes"
Apr 16 16:50:20.003536 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:20.003494    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf" podUID="3d711f7c-da57-4df7-bbf0-19670ac3b317" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Apr 16 16:50:30.003253 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:30.003169    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf" podUID="3d711f7c-da57-4df7-bbf0-19670ac3b317" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Apr 16 16:50:40.003417 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:40.003383    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf"
Apr 16 16:50:42.702338 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:42.702306    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf"]
Apr 16 16:50:42.702727 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:42.702530    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf" podUID="3d711f7c-da57-4df7-bbf0-19670ac3b317" containerName="kserve-container" containerID="cri-o://9812db8729bdd0fd25bd826765e44573c337afeca6dc69d67e4e5f57037fd78e" gracePeriod=30
Apr 16 16:50:42.801731 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:42.801704    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45"]
Apr 16 16:50:42.802003 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:42.801992    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" containerName="storage-initializer"
Apr 16 16:50:42.802047 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:42.802004    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" containerName="storage-initializer"
Apr 16 16:50:42.802047 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:42.802022    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" containerName="kserve-container"
Apr 16 16:50:42.802047 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:42.802027    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" containerName="kserve-container"
Apr 16 16:50:42.802138 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:42.802049    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" containerName="kserve-agent"
Apr 16 16:50:42.802138 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:42.802055    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" containerName="kserve-agent"
Apr 16 16:50:42.802138 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:42.802119    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" containerName="kserve-container"
Apr 16 16:50:42.802138 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:42.802131    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="5bea9688-1f70-49bd-b046-e68c69ffd3bf" containerName="kserve-agent"
Apr 16 16:50:42.805055 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:42.805038    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45"
Apr 16 16:50:42.813753 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:42.813730    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45"]
Apr 16 16:50:42.922131 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:42.922102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18a28a35-f087-4b61-b890-edaaa4f3df28-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-86b49c4466-mjc45\" (UID: \"18a28a35-f087-4b61-b890-edaaa4f3df28\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45"
Apr 16 16:50:43.023369 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:43.023314    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18a28a35-f087-4b61-b890-edaaa4f3df28-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-86b49c4466-mjc45\" (UID: \"18a28a35-f087-4b61-b890-edaaa4f3df28\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45"
Apr 16 16:50:43.023664 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:43.023647    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18a28a35-f087-4b61-b890-edaaa4f3df28-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-86b49c4466-mjc45\" (UID: \"18a28a35-f087-4b61-b890-edaaa4f3df28\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45"
Apr 16 16:50:43.115380 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:43.115360    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45"
Apr 16 16:50:43.228532 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:43.228505    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45"]
Apr 16 16:50:43.231893 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:50:43.231861    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a28a35_f087_4b61_b890_edaaa4f3df28.slice/crio-4e50bd893a33b6557f79ff4518ba3a5ce777ee55221c1719425127a0d806bbbb WatchSource:0}: Error finding container 4e50bd893a33b6557f79ff4518ba3a5ce777ee55221c1719425127a0d806bbbb: Status 404 returned error can't find the container with id 4e50bd893a33b6557f79ff4518ba3a5ce777ee55221c1719425127a0d806bbbb
Apr 16 16:50:44.126295 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:44.126261    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45" event={"ID":"18a28a35-f087-4b61-b890-edaaa4f3df28","Type":"ContainerStarted","Data":"834c2e18931aa1150ca4c94b2f8bf63929f00afee3d84882eed648e42c490f07"}
Apr 16 16:50:44.126295 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:44.126298    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45" event={"ID":"18a28a35-f087-4b61-b890-edaaa4f3df28","Type":"ContainerStarted","Data":"4e50bd893a33b6557f79ff4518ba3a5ce777ee55221c1719425127a0d806bbbb"}
Apr 16 16:50:45.035432 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:45.035411    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf"
Apr 16 16:50:45.130511 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:45.130443    2578 generic.go:358] "Generic (PLEG): container finished" podID="3d711f7c-da57-4df7-bbf0-19670ac3b317" containerID="9812db8729bdd0fd25bd826765e44573c337afeca6dc69d67e4e5f57037fd78e" exitCode=0
Apr 16 16:50:45.130511 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:45.130500    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf"
Apr 16 16:50:45.130910 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:45.130527    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf" event={"ID":"3d711f7c-da57-4df7-bbf0-19670ac3b317","Type":"ContainerDied","Data":"9812db8729bdd0fd25bd826765e44573c337afeca6dc69d67e4e5f57037fd78e"}
Apr 16 16:50:45.130910 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:45.130581    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf" event={"ID":"3d711f7c-da57-4df7-bbf0-19670ac3b317","Type":"ContainerDied","Data":"93c18d032b90e419972118dc1356f2d1b48d62ac70366ee0a91a5b41e98fde69"}
Apr 16 16:50:45.130910 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:45.130596    2578 scope.go:117] "RemoveContainer" containerID="9812db8729bdd0fd25bd826765e44573c337afeca6dc69d67e4e5f57037fd78e"
Apr 16 16:50:45.138125 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:45.138106    2578 scope.go:117] "RemoveContainer" containerID="0eba1d9aff6fb8d35ef91804676a4175ff77916a476d59f08fdad43b68fbe6ba"
Apr 16 16:50:45.142142 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:45.142123    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d711f7c-da57-4df7-bbf0-19670ac3b317-kserve-provision-location\") pod \"3d711f7c-da57-4df7-bbf0-19670ac3b317\" (UID: \"3d711f7c-da57-4df7-bbf0-19670ac3b317\") "
Apr 16 16:50:45.144771 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:45.144756    2578 scope.go:117] "RemoveContainer" containerID="9812db8729bdd0fd25bd826765e44573c337afeca6dc69d67e4e5f57037fd78e"
Apr 16 16:50:45.145025 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:50:45.145003    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9812db8729bdd0fd25bd826765e44573c337afeca6dc69d67e4e5f57037fd78e\": container with ID starting with 9812db8729bdd0fd25bd826765e44573c337afeca6dc69d67e4e5f57037fd78e not found: ID does not exist" containerID="9812db8729bdd0fd25bd826765e44573c337afeca6dc69d67e4e5f57037fd78e"
Apr 16 16:50:45.145085 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:45.145036    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9812db8729bdd0fd25bd826765e44573c337afeca6dc69d67e4e5f57037fd78e"} err="failed to get container status \"9812db8729bdd0fd25bd826765e44573c337afeca6dc69d67e4e5f57037fd78e\": rpc error: code = NotFound desc = could not find container \"9812db8729bdd0fd25bd826765e44573c337afeca6dc69d67e4e5f57037fd78e\": container with ID starting with 9812db8729bdd0fd25bd826765e44573c337afeca6dc69d67e4e5f57037fd78e not found: ID does not exist"
Apr 16 16:50:45.145085 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:45.145059    2578 scope.go:117] "RemoveContainer" containerID="0eba1d9aff6fb8d35ef91804676a4175ff77916a476d59f08fdad43b68fbe6ba"
Apr 16 16:50:45.145268 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:50:45.145251    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eba1d9aff6fb8d35ef91804676a4175ff77916a476d59f08fdad43b68fbe6ba\": container with ID starting with 0eba1d9aff6fb8d35ef91804676a4175ff77916a476d59f08fdad43b68fbe6ba not found: ID does not exist" containerID="0eba1d9aff6fb8d35ef91804676a4175ff77916a476d59f08fdad43b68fbe6ba"
Apr 16 16:50:45.145309 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:45.145274    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eba1d9aff6fb8d35ef91804676a4175ff77916a476d59f08fdad43b68fbe6ba"} err="failed to get container status \"0eba1d9aff6fb8d35ef91804676a4175ff77916a476d59f08fdad43b68fbe6ba\": rpc error: code = NotFound desc = could not find container \"0eba1d9aff6fb8d35ef91804676a4175ff77916a476d59f08fdad43b68fbe6ba\": container with ID starting with 0eba1d9aff6fb8d35ef91804676a4175ff77916a476d59f08fdad43b68fbe6ba not found: ID does not exist"
Apr 16 16:50:45.152268 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:45.152247    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d711f7c-da57-4df7-bbf0-19670ac3b317-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d711f7c-da57-4df7-bbf0-19670ac3b317" (UID: "3d711f7c-da57-4df7-bbf0-19670ac3b317"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:50:45.243273 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:45.243254    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d711f7c-da57-4df7-bbf0-19670ac3b317-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:50:45.445829 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:45.445768    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf"]
Apr 16 16:50:45.449728 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:45.449708    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-288rf"]
Apr 16 16:50:47.235123 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:47.235089    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d711f7c-da57-4df7-bbf0-19670ac3b317" path="/var/lib/kubelet/pods/3d711f7c-da57-4df7-bbf0-19670ac3b317/volumes"
Apr 16 16:50:48.140565 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:48.140536    2578 generic.go:358] "Generic (PLEG): container finished" podID="18a28a35-f087-4b61-b890-edaaa4f3df28" containerID="834c2e18931aa1150ca4c94b2f8bf63929f00afee3d84882eed648e42c490f07" exitCode=0
Apr 16 16:50:48.140684 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:48.140612    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45" event={"ID":"18a28a35-f087-4b61-b890-edaaa4f3df28","Type":"ContainerDied","Data":"834c2e18931aa1150ca4c94b2f8bf63929f00afee3d84882eed648e42c490f07"}
Apr 16 16:50:49.150901 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:49.150866    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45" event={"ID":"18a28a35-f087-4b61-b890-edaaa4f3df28","Type":"ContainerStarted","Data":"6da5f1f0c85d86640cbc319250cdff9c78b4f69dbe9ad1a5c1cb2dce754c6652"}
Apr 16 16:50:49.151268 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:49.151143    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45"
Apr 16 16:50:49.152284 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:49.152260    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45" podUID="18a28a35-f087-4b61-b890-edaaa4f3df28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Apr 16 16:50:49.169604 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:49.169548    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45" podStartSLOduration=7.169536253 podStartE2EDuration="7.169536253s" podCreationTimestamp="2026-04-16 16:50:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:50:49.167388771 +0000 UTC m=+1258.564605159" watchObservedRunningTime="2026-04-16 16:50:49.169536253 +0000 UTC m=+1258.566752641"
Apr 16 16:50:50.154311 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:50:50.154279    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45" podUID="18a28a35-f087-4b61-b890-edaaa4f3df28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Apr 16 16:51:00.155265 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:00.155225    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45" podUID="18a28a35-f087-4b61-b890-edaaa4f3df28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Apr 16 16:51:10.154646 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:10.154600    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45" podUID="18a28a35-f087-4b61-b890-edaaa4f3df28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Apr 16 16:51:20.154699 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:20.154660    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45" podUID="18a28a35-f087-4b61-b890-edaaa4f3df28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Apr 16 16:51:30.155474 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:30.155447    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45"
Apr 16 16:51:34.272290 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:34.272261    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45"]
Apr 16 16:51:34.272743 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:34.272688    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45" podUID="18a28a35-f087-4b61-b890-edaaa4f3df28" containerName="kserve-container" containerID="cri-o://6da5f1f0c85d86640cbc319250cdff9c78b4f69dbe9ad1a5c1cb2dce754c6652" gracePeriod=30
Apr 16 16:51:34.334076 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:34.334048    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d"]
Apr 16 16:51:34.334361 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:34.334349    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d711f7c-da57-4df7-bbf0-19670ac3b317" containerName="kserve-container"
Apr 16 16:51:34.334413 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:34.334363    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d711f7c-da57-4df7-bbf0-19670ac3b317" containerName="kserve-container"
Apr 16 16:51:34.334413 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:34.334376    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d711f7c-da57-4df7-bbf0-19670ac3b317" containerName="storage-initializer"
Apr 16 16:51:34.334413 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:34.334381    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d711f7c-da57-4df7-bbf0-19670ac3b317" containerName="storage-initializer"
Apr 16 16:51:34.334503 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:34.334430    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="3d711f7c-da57-4df7-bbf0-19670ac3b317" containerName="kserve-container"
Apr 16 16:51:34.337572 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:34.337544    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d"
Apr 16 16:51:34.346245 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:34.346217    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d"]
Apr 16 16:51:34.381196 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:34.381170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b223d50e-cb23-43fb-ba00-678704c85aa6-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d\" (UID: \"b223d50e-cb23-43fb-ba00-678704c85aa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d"
Apr 16 16:51:34.481709 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:34.481686    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b223d50e-cb23-43fb-ba00-678704c85aa6-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d\" (UID: \"b223d50e-cb23-43fb-ba00-678704c85aa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d"
Apr 16 16:51:34.482005 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:34.481990    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b223d50e-cb23-43fb-ba00-678704c85aa6-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d\" (UID: \"b223d50e-cb23-43fb-ba00-678704c85aa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d"
Apr 16 16:51:34.647948 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:34.647871    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d"
Apr 16 16:51:34.762947 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:34.762920    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d"]
Apr 16 16:51:34.766444 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:51:34.766411    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb223d50e_cb23_43fb_ba00_678704c85aa6.slice/crio-5098534f3cc6b93ea7512b2a6912b3f0cfb4f39be8710454e37dcadbe1f04936 WatchSource:0}: Error finding container 5098534f3cc6b93ea7512b2a6912b3f0cfb4f39be8710454e37dcadbe1f04936: Status 404 returned error can't find the container with id 5098534f3cc6b93ea7512b2a6912b3f0cfb4f39be8710454e37dcadbe1f04936
Apr 16 16:51:35.286648 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:35.286612    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d" event={"ID":"b223d50e-cb23-43fb-ba00-678704c85aa6","Type":"ContainerStarted","Data":"dc24fbce77f0b6a209ac67f0abfa9cf0268fb385a4bcb0b3e05d2ce41cd54e89"}
Apr 16 16:51:35.286648 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:35.286649    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d" event={"ID":"b223d50e-cb23-43fb-ba00-678704c85aa6","Type":"ContainerStarted","Data":"5098534f3cc6b93ea7512b2a6912b3f0cfb4f39be8710454e37dcadbe1f04936"}
Apr 16 16:51:36.716241 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:36.716220    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45"
Apr 16 16:51:36.802230 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:36.802170    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18a28a35-f087-4b61-b890-edaaa4f3df28-kserve-provision-location\") pod \"18a28a35-f087-4b61-b890-edaaa4f3df28\" (UID: \"18a28a35-f087-4b61-b890-edaaa4f3df28\") "
Apr 16 16:51:36.811328 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:36.811304    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a28a35-f087-4b61-b890-edaaa4f3df28-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "18a28a35-f087-4b61-b890-edaaa4f3df28" (UID: "18a28a35-f087-4b61-b890-edaaa4f3df28"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:51:36.903307 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:36.903282    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18a28a35-f087-4b61-b890-edaaa4f3df28-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:51:37.293655 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:37.293624    2578 generic.go:358] "Generic (PLEG): container finished" podID="18a28a35-f087-4b61-b890-edaaa4f3df28" containerID="6da5f1f0c85d86640cbc319250cdff9c78b4f69dbe9ad1a5c1cb2dce754c6652" exitCode=0
Apr 16 16:51:37.293762 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:37.293671    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45" event={"ID":"18a28a35-f087-4b61-b890-edaaa4f3df28","Type":"ContainerDied","Data":"6da5f1f0c85d86640cbc319250cdff9c78b4f69dbe9ad1a5c1cb2dce754c6652"}
Apr 16 16:51:37.293762 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:37.293693    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45"
Apr 16 16:51:37.293762 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:37.293708    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45" event={"ID":"18a28a35-f087-4b61-b890-edaaa4f3df28","Type":"ContainerDied","Data":"4e50bd893a33b6557f79ff4518ba3a5ce777ee55221c1719425127a0d806bbbb"}
Apr 16 16:51:37.293762 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:37.293725    2578 scope.go:117] "RemoveContainer" containerID="6da5f1f0c85d86640cbc319250cdff9c78b4f69dbe9ad1a5c1cb2dce754c6652"
Apr 16 16:51:37.301119 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:37.301102    2578 scope.go:117] "RemoveContainer" containerID="834c2e18931aa1150ca4c94b2f8bf63929f00afee3d84882eed648e42c490f07"
Apr 16 16:51:37.308028 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:37.308009    2578 scope.go:117] "RemoveContainer" containerID="6da5f1f0c85d86640cbc319250cdff9c78b4f69dbe9ad1a5c1cb2dce754c6652"
Apr 16 16:51:37.308269 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:51:37.308248    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da5f1f0c85d86640cbc319250cdff9c78b4f69dbe9ad1a5c1cb2dce754c6652\": container with ID starting with 6da5f1f0c85d86640cbc319250cdff9c78b4f69dbe9ad1a5c1cb2dce754c6652 not found: ID does not exist" containerID="6da5f1f0c85d86640cbc319250cdff9c78b4f69dbe9ad1a5c1cb2dce754c6652"
Apr 16 16:51:37.308332 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:37.308280    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da5f1f0c85d86640cbc319250cdff9c78b4f69dbe9ad1a5c1cb2dce754c6652"} err="failed to get container status \"6da5f1f0c85d86640cbc319250cdff9c78b4f69dbe9ad1a5c1cb2dce754c6652\": rpc error: code = NotFound desc = could not find container \"6da5f1f0c85d86640cbc319250cdff9c78b4f69dbe9ad1a5c1cb2dce754c6652\": container with ID starting with 6da5f1f0c85d86640cbc319250cdff9c78b4f69dbe9ad1a5c1cb2dce754c6652 not found: ID does not exist"
Apr 16 16:51:37.308332 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:37.308298    2578 scope.go:117] "RemoveContainer" containerID="834c2e18931aa1150ca4c94b2f8bf63929f00afee3d84882eed648e42c490f07"
Apr 16 16:51:37.308578 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:51:37.308536    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834c2e18931aa1150ca4c94b2f8bf63929f00afee3d84882eed648e42c490f07\": container with ID starting with 834c2e18931aa1150ca4c94b2f8bf63929f00afee3d84882eed648e42c490f07 not found: ID does not exist" containerID="834c2e18931aa1150ca4c94b2f8bf63929f00afee3d84882eed648e42c490f07"
Apr 16 16:51:37.308643 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:37.308584    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834c2e18931aa1150ca4c94b2f8bf63929f00afee3d84882eed648e42c490f07"} err="failed to get container status \"834c2e18931aa1150ca4c94b2f8bf63929f00afee3d84882eed648e42c490f07\": rpc error: code = NotFound desc = could not find container \"834c2e18931aa1150ca4c94b2f8bf63929f00afee3d84882eed648e42c490f07\": container with ID starting with 834c2e18931aa1150ca4c94b2f8bf63929f00afee3d84882eed648e42c490f07 not found: ID does not exist"
Apr 16 16:51:37.308819 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:37.308800    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45"]
Apr 16 16:51:37.315100 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:37.315080    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-mjc45"]
Apr 16 16:51:39.235439 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:39.235412    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18a28a35-f087-4b61-b890-edaaa4f3df28" path="/var/lib/kubelet/pods/18a28a35-f087-4b61-b890-edaaa4f3df28/volumes"
Apr 16 16:51:39.302345 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:39.302319    2578 generic.go:358] "Generic (PLEG): container finished" podID="b223d50e-cb23-43fb-ba00-678704c85aa6" containerID="dc24fbce77f0b6a209ac67f0abfa9cf0268fb385a4bcb0b3e05d2ce41cd54e89" exitCode=0
Apr 16 16:51:39.302458 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:39.302384    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d" event={"ID":"b223d50e-cb23-43fb-ba00-678704c85aa6","Type":"ContainerDied","Data":"dc24fbce77f0b6a209ac67f0abfa9cf0268fb385a4bcb0b3e05d2ce41cd54e89"}
Apr 16 16:51:40.306873 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:40.306840    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d" event={"ID":"b223d50e-cb23-43fb-ba00-678704c85aa6","Type":"ContainerStarted","Data":"8f3cf4628c9750999f1e07bfa86781e505cc23ea09a4ccc4e055fc1956616c19"}
Apr 16 16:51:40.307268 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:40.307224    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d"
Apr 16 16:51:40.308362 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:40.308332    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d" podUID="b223d50e-cb23-43fb-ba00-678704c85aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Apr 16 16:51:40.321719 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:40.321685    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d" podStartSLOduration=6.321673117 podStartE2EDuration="6.321673117s" podCreationTimestamp="2026-04-16 16:51:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:51:40.320450426 +0000 UTC m=+1309.717666813" watchObservedRunningTime="2026-04-16 16:51:40.321673117 +0000 UTC m=+1309.718889504"
Apr 16 16:51:41.310122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:41.310087    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d" podUID="b223d50e-cb23-43fb-ba00-678704c85aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Apr 16 16:51:51.310054 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:51:51.310017    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d" podUID="b223d50e-cb23-43fb-ba00-678704c85aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Apr 16 16:52:01.310454 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:01.310416    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d" podUID="b223d50e-cb23-43fb-ba00-678704c85aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Apr 16 16:52:11.310959 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:11.310919    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d" podUID="b223d50e-cb23-43fb-ba00-678704c85aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Apr 16 16:52:21.311715 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:21.311681    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d"
Apr 16 16:52:26.082778 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:26.082741    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d"]
Apr 16 16:52:26.083247 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:26.083073    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d" podUID="b223d50e-cb23-43fb-ba00-678704c85aa6" containerName="kserve-container" containerID="cri-o://8f3cf4628c9750999f1e07bfa86781e505cc23ea09a4ccc4e055fc1956616c19" gracePeriod=30
Apr 16 16:52:26.176291 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:26.176264    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l"]
Apr 16 16:52:26.176612 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:26.176596    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a28a35-f087-4b61-b890-edaaa4f3df28" containerName="storage-initializer"
Apr 16 16:52:26.176612 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:26.176611    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a28a35-f087-4b61-b890-edaaa4f3df28" containerName="storage-initializer"
Apr 16 16:52:26.176740 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:26.176628    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a28a35-f087-4b61-b890-edaaa4f3df28" containerName="kserve-container"
Apr 16 16:52:26.176740 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:26.176650    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a28a35-f087-4b61-b890-edaaa4f3df28" containerName="kserve-container"
Apr 16 16:52:26.176740 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:26.176702    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="18a28a35-f087-4b61-b890-edaaa4f3df28" containerName="kserve-container"
Apr 16 16:52:26.179689 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:26.179669    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l"
Apr 16 16:52:26.185763 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:26.185742    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l"]
Apr 16 16:52:26.354444 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:26.354358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a588ca72-f1f1-4158-afee-35c9012408fc-kserve-provision-location\") pod \"isvc-pmml-predictor-5584ffd8c9-qs45l\" (UID: \"a588ca72-f1f1-4158-afee-35c9012408fc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l"
Apr 16 16:52:26.454903 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:26.454880    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a588ca72-f1f1-4158-afee-35c9012408fc-kserve-provision-location\") pod \"isvc-pmml-predictor-5584ffd8c9-qs45l\" (UID: \"a588ca72-f1f1-4158-afee-35c9012408fc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l"
Apr 16 16:52:26.455187 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:26.455170    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a588ca72-f1f1-4158-afee-35c9012408fc-kserve-provision-location\") pod \"isvc-pmml-predictor-5584ffd8c9-qs45l\" (UID: \"a588ca72-f1f1-4158-afee-35c9012408fc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l"
Apr 16 16:52:26.490410 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:26.490391    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l"
Apr 16 16:52:26.605538 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:26.605351    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l"]
Apr 16 16:52:26.608322 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:52:26.608294    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda588ca72_f1f1_4158_afee_35c9012408fc.slice/crio-cecbeee8b55450f9e34019370d755354956728fd57bdec4bdcfe86c6356955bf WatchSource:0}: Error finding container cecbeee8b55450f9e34019370d755354956728fd57bdec4bdcfe86c6356955bf: Status 404 returned error can't find the container with id cecbeee8b55450f9e34019370d755354956728fd57bdec4bdcfe86c6356955bf
Apr 16 16:52:27.435856 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:27.435820    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" event={"ID":"a588ca72-f1f1-4158-afee-35c9012408fc","Type":"ContainerStarted","Data":"3f7e18b763cfbc758706450b2d9ed3f845b3348f6fd9c3ec9a8efa469ce55364"}
Apr 16 16:52:27.435856 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:27.435856    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" event={"ID":"a588ca72-f1f1-4158-afee-35c9012408fc","Type":"ContainerStarted","Data":"cecbeee8b55450f9e34019370d755354956728fd57bdec4bdcfe86c6356955bf"}
Apr 16 16:52:28.440420 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:28.440392    2578 generic.go:358] "Generic (PLEG): container finished" podID="b223d50e-cb23-43fb-ba00-678704c85aa6" containerID="8f3cf4628c9750999f1e07bfa86781e505cc23ea09a4ccc4e055fc1956616c19" exitCode=0
Apr 16 16:52:28.440855 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:28.440454    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d" event={"ID":"b223d50e-cb23-43fb-ba00-678704c85aa6","Type":"ContainerDied","Data":"8f3cf4628c9750999f1e07bfa86781e505cc23ea09a4ccc4e055fc1956616c19"}
Apr 16 16:52:28.520630 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:28.520611    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d"
Apr 16 16:52:28.671841 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:28.671781    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b223d50e-cb23-43fb-ba00-678704c85aa6-kserve-provision-location\") pod \"b223d50e-cb23-43fb-ba00-678704c85aa6\" (UID: \"b223d50e-cb23-43fb-ba00-678704c85aa6\") "
Apr 16 16:52:28.681796 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:28.681770    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b223d50e-cb23-43fb-ba00-678704c85aa6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b223d50e-cb23-43fb-ba00-678704c85aa6" (UID: "b223d50e-cb23-43fb-ba00-678704c85aa6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:52:28.772816 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:28.772793    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b223d50e-cb23-43fb-ba00-678704c85aa6-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:52:29.444984 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:29.444954    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d" event={"ID":"b223d50e-cb23-43fb-ba00-678704c85aa6","Type":"ContainerDied","Data":"5098534f3cc6b93ea7512b2a6912b3f0cfb4f39be8710454e37dcadbe1f04936"}
Apr 16 16:52:29.444984 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:29.444969    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d"
Apr 16 16:52:29.445410 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:29.444999    2578 scope.go:117] "RemoveContainer" containerID="8f3cf4628c9750999f1e07bfa86781e505cc23ea09a4ccc4e055fc1956616c19"
Apr 16 16:52:29.452739 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:29.452724    2578 scope.go:117] "RemoveContainer" containerID="dc24fbce77f0b6a209ac67f0abfa9cf0268fb385a4bcb0b3e05d2ce41cd54e89"
Apr 16 16:52:29.461417 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:29.461398    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d"]
Apr 16 16:52:29.465402 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:29.465383    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-7mj5d"]
Apr 16 16:52:31.234570 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:31.234535    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b223d50e-cb23-43fb-ba00-678704c85aa6" path="/var/lib/kubelet/pods/b223d50e-cb23-43fb-ba00-678704c85aa6/volumes"
Apr 16 16:52:31.452746 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:31.452714    2578 generic.go:358] "Generic (PLEG): container finished" podID="a588ca72-f1f1-4158-afee-35c9012408fc" containerID="3f7e18b763cfbc758706450b2d9ed3f845b3348f6fd9c3ec9a8efa469ce55364" exitCode=0
Apr 16 16:52:31.452905 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:31.452770    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" event={"ID":"a588ca72-f1f1-4158-afee-35c9012408fc","Type":"ContainerDied","Data":"3f7e18b763cfbc758706450b2d9ed3f845b3348f6fd9c3ec9a8efa469ce55364"}
Apr 16 16:52:31.453773 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:31.453754    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 16:52:38.481337 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:38.481308    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" event={"ID":"a588ca72-f1f1-4158-afee-35c9012408fc","Type":"ContainerStarted","Data":"7367ea0758dbb9a612539764e94ba97e561792c5b2c026777402f96d891bbf9a"}
Apr 16 16:52:38.481729 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:38.481542    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l"
Apr 16 16:52:38.482958 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:38.482932    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" podUID="a588ca72-f1f1-4158-afee-35c9012408fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Apr 16 16:52:38.496904 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:38.496859    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" podStartSLOduration=5.826914473 podStartE2EDuration="12.496848223s" podCreationTimestamp="2026-04-16 16:52:26 +0000 UTC" firstStartedPulling="2026-04-16 16:52:31.453907123 +0000 UTC m=+1360.851123488" lastFinishedPulling="2026-04-16 16:52:38.123840868 +0000 UTC m=+1367.521057238" observedRunningTime="2026-04-16 16:52:38.495375992 +0000 UTC m=+1367.892592392" watchObservedRunningTime="2026-04-16 16:52:38.496848223 +0000 UTC m=+1367.894064611"
Apr 16 16:52:39.484997 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:39.484960    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" podUID="a588ca72-f1f1-4158-afee-35c9012408fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Apr 16 16:52:49.484987 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:49.484948    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" podUID="a588ca72-f1f1-4158-afee-35c9012408fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Apr 16 16:52:59.485102 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:52:59.485060    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" podUID="a588ca72-f1f1-4158-afee-35c9012408fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Apr 16 16:53:09.485741 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:09.485699    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" podUID="a588ca72-f1f1-4158-afee-35c9012408fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Apr 16 16:53:19.486005 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:19.485964    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" podUID="a588ca72-f1f1-4158-afee-35c9012408fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Apr 16 16:53:29.485693 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:29.485610    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" podUID="a588ca72-f1f1-4158-afee-35c9012408fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Apr 16 16:53:39.485621 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:39.485583    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" podUID="a588ca72-f1f1-4158-afee-35c9012408fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Apr 16 16:53:45.231599 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:45.231537    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" podUID="a588ca72-f1f1-4158-afee-35c9012408fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Apr 16 16:53:55.235647 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:55.235621    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l"
Apr 16 16:53:57.785869 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:57.785833    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l"]
Apr 16 16:53:57.786358 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:57.786173    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" podUID="a588ca72-f1f1-4158-afee-35c9012408fc" containerName="kserve-container" containerID="cri-o://7367ea0758dbb9a612539764e94ba97e561792c5b2c026777402f96d891bbf9a" gracePeriod=30
Apr 16 16:53:57.877756 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:57.877722    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq"]
Apr 16 16:53:57.878615 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:57.878592    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b223d50e-cb23-43fb-ba00-678704c85aa6" containerName="storage-initializer"
Apr 16 16:53:57.878770 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:57.878758    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b223d50e-cb23-43fb-ba00-678704c85aa6" containerName="storage-initializer"
Apr 16 16:53:57.878983 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:57.878971    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b223d50e-cb23-43fb-ba00-678704c85aa6" containerName="kserve-container"
Apr 16 16:53:57.879068 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:57.879059    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b223d50e-cb23-43fb-ba00-678704c85aa6" containerName="kserve-container"
Apr 16 16:53:57.879278 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:57.879266    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="b223d50e-cb23-43fb-ba00-678704c85aa6" containerName="kserve-container"
Apr 16 16:53:57.883191 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:57.883164    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq"
Apr 16 16:53:57.886305 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:57.886281    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq"]
Apr 16 16:53:58.018879 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:58.018847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/319081fc-5b36-494b-82d5-2021e1f9b528-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-7576f6b69f-9l9zq\" (UID: \"319081fc-5b36-494b-82d5-2021e1f9b528\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq"
Apr 16 16:53:58.120121 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:58.120051    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/319081fc-5b36-494b-82d5-2021e1f9b528-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-7576f6b69f-9l9zq\" (UID: \"319081fc-5b36-494b-82d5-2021e1f9b528\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq"
Apr 16 16:53:58.120401 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:58.120383    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/319081fc-5b36-494b-82d5-2021e1f9b528-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-7576f6b69f-9l9zq\" (UID: \"319081fc-5b36-494b-82d5-2021e1f9b528\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq"
Apr 16 16:53:58.193719 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:58.193694    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq"
Apr 16 16:53:58.311001 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:58.310905    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq"]
Apr 16 16:53:58.314292 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:53:58.314265    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319081fc_5b36_494b_82d5_2021e1f9b528.slice/crio-8077ba5e46caf004da67e521a3c2f0e1ba581c6321784822d408507dccc96c9c WatchSource:0}: Error finding container 8077ba5e46caf004da67e521a3c2f0e1ba581c6321784822d408507dccc96c9c: Status 404 returned error can't find the container with id 8077ba5e46caf004da67e521a3c2f0e1ba581c6321784822d408507dccc96c9c
Apr 16 16:53:58.710712 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:58.710672    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq" event={"ID":"319081fc-5b36-494b-82d5-2021e1f9b528","Type":"ContainerStarted","Data":"7b82e0e1c4a50ef911ebb1db48b39b4fee73296ee2bd23ba09e65c8caeac3ccb"}
Apr 16 16:53:58.710712 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:53:58.710713    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq" event={"ID":"319081fc-5b36-494b-82d5-2021e1f9b528","Type":"ContainerStarted","Data":"8077ba5e46caf004da67e521a3c2f0e1ba581c6321784822d408507dccc96c9c"}
Apr 16 16:54:00.720001 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:00.719971    2578 generic.go:358] "Generic (PLEG): container finished" podID="a588ca72-f1f1-4158-afee-35c9012408fc" containerID="7367ea0758dbb9a612539764e94ba97e561792c5b2c026777402f96d891bbf9a" exitCode=0
Apr 16 16:54:00.720315 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:00.720023    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" event={"ID":"a588ca72-f1f1-4158-afee-35c9012408fc","Type":"ContainerDied","Data":"7367ea0758dbb9a612539764e94ba97e561792c5b2c026777402f96d891bbf9a"}
Apr 16 16:54:01.012772 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:01.012748    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l"
Apr 16 16:54:01.042319 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:01.042291    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a588ca72-f1f1-4158-afee-35c9012408fc-kserve-provision-location\") pod \"a588ca72-f1f1-4158-afee-35c9012408fc\" (UID: \"a588ca72-f1f1-4158-afee-35c9012408fc\") "
Apr 16 16:54:01.042627 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:01.042605    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a588ca72-f1f1-4158-afee-35c9012408fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a588ca72-f1f1-4158-afee-35c9012408fc" (UID: "a588ca72-f1f1-4158-afee-35c9012408fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:54:01.143286 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:01.143261    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a588ca72-f1f1-4158-afee-35c9012408fc-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:54:01.724684 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:01.724648    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l" event={"ID":"a588ca72-f1f1-4158-afee-35c9012408fc","Type":"ContainerDied","Data":"cecbeee8b55450f9e34019370d755354956728fd57bdec4bdcfe86c6356955bf"}
Apr 16 16:54:01.724684 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:01.724670    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l"
Apr 16 16:54:01.725186 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:01.724693    2578 scope.go:117] "RemoveContainer" containerID="7367ea0758dbb9a612539764e94ba97e561792c5b2c026777402f96d891bbf9a"
Apr 16 16:54:01.732259 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:01.732242    2578 scope.go:117] "RemoveContainer" containerID="3f7e18b763cfbc758706450b2d9ed3f845b3348f6fd9c3ec9a8efa469ce55364"
Apr 16 16:54:01.741445 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:01.741421    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l"]
Apr 16 16:54:01.745437 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:01.745418    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-qs45l"]
Apr 16 16:54:02.729355 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:02.729319    2578 generic.go:358] "Generic (PLEG): container finished" podID="319081fc-5b36-494b-82d5-2021e1f9b528" containerID="7b82e0e1c4a50ef911ebb1db48b39b4fee73296ee2bd23ba09e65c8caeac3ccb" exitCode=0
Apr 16 16:54:02.729866 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:02.729397    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq" event={"ID":"319081fc-5b36-494b-82d5-2021e1f9b528","Type":"ContainerDied","Data":"7b82e0e1c4a50ef911ebb1db48b39b4fee73296ee2bd23ba09e65c8caeac3ccb"}
Apr 16 16:54:03.235723 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:03.235695    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a588ca72-f1f1-4158-afee-35c9012408fc" path="/var/lib/kubelet/pods/a588ca72-f1f1-4158-afee-35c9012408fc/volumes"
Apr 16 16:54:03.735613 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:03.735581    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq" event={"ID":"319081fc-5b36-494b-82d5-2021e1f9b528","Type":"ContainerStarted","Data":"6706108dbd936be315704399f4e487053ec7a3ea57b1eb925f6640074f8a0fd2"}
Apr 16 16:54:03.735990 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:03.735888    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq"
Apr 16 16:54:03.737239 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:03.737210    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq" podUID="319081fc-5b36-494b-82d5-2021e1f9b528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Apr 16 16:54:03.751772 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:03.751728    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq" podStartSLOduration=6.7517135889999995 podStartE2EDuration="6.751713589s" podCreationTimestamp="2026-04-16 16:53:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:54:03.750313236 +0000 UTC m=+1453.147529635" watchObservedRunningTime="2026-04-16 16:54:03.751713589 +0000 UTC m=+1453.148929979"
Apr 16 16:54:04.738991 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:04.738952    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq" podUID="319081fc-5b36-494b-82d5-2021e1f9b528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Apr 16 16:54:14.738918 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:14.738876    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq" podUID="319081fc-5b36-494b-82d5-2021e1f9b528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Apr 16 16:54:24.739424 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:24.739379    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq" podUID="319081fc-5b36-494b-82d5-2021e1f9b528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Apr 16 16:54:34.739459 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:34.739418    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq" podUID="319081fc-5b36-494b-82d5-2021e1f9b528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Apr 16 16:54:44.739835 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:44.739792    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq" podUID="319081fc-5b36-494b-82d5-2021e1f9b528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Apr 16 16:54:51.225265 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:51.225236    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 16:54:51.227270 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:51.227244    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 16:54:54.738944 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:54:54.738905    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq" podUID="319081fc-5b36-494b-82d5-2021e1f9b528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Apr 16 16:55:04.739988 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:04.739948    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq" podUID="319081fc-5b36-494b-82d5-2021e1f9b528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Apr 16 16:55:14.740743 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:14.740710    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq"
Apr 16 16:55:18.877005 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:18.876976    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq"]
Apr 16 16:55:18.877365 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:18.877205    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq" podUID="319081fc-5b36-494b-82d5-2021e1f9b528" containerName="kserve-container" containerID="cri-o://6706108dbd936be315704399f4e487053ec7a3ea57b1eb925f6640074f8a0fd2" gracePeriod=30
Apr 16 16:55:18.927372 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:18.927346    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7"]
Apr 16 16:55:18.927673 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:18.927660    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a588ca72-f1f1-4158-afee-35c9012408fc" containerName="kserve-container"
Apr 16 16:55:18.927730 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:18.927674    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a588ca72-f1f1-4158-afee-35c9012408fc" containerName="kserve-container"
Apr 16 16:55:18.927730 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:18.927690    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a588ca72-f1f1-4158-afee-35c9012408fc" containerName="storage-initializer"
Apr 16 16:55:18.927730 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:18.927696    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a588ca72-f1f1-4158-afee-35c9012408fc" containerName="storage-initializer"
Apr 16 16:55:18.927827 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:18.927746    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a588ca72-f1f1-4158-afee-35c9012408fc" containerName="kserve-container"
Apr 16 16:55:18.930568 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:18.930537    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7"
Apr 16 16:55:18.938744 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:18.938723    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7"]
Apr 16 16:55:19.069856 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:19.069826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc5f7cc7-9d96-4368-af29-e2c53b22bbd8-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7\" (UID: \"fc5f7cc7-9d96-4368-af29-e2c53b22bbd8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7"
Apr 16 16:55:19.171228 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:19.171161    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc5f7cc7-9d96-4368-af29-e2c53b22bbd8-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7\" (UID: \"fc5f7cc7-9d96-4368-af29-e2c53b22bbd8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7"
Apr 16 16:55:19.171484 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:19.171469    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc5f7cc7-9d96-4368-af29-e2c53b22bbd8-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7\" (UID: \"fc5f7cc7-9d96-4368-af29-e2c53b22bbd8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7"
Apr 16 16:55:19.241110 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:19.241085    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7"
Apr 16 16:55:19.355364 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:19.355337    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7"]
Apr 16 16:55:19.358494 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:55:19.358465    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5f7cc7_9d96_4368_af29_e2c53b22bbd8.slice/crio-40cbb6d647b2a5b23bfddfe6b44f919c48018b47b8582767b75eb405226f11c2 WatchSource:0}: Error finding container 40cbb6d647b2a5b23bfddfe6b44f919c48018b47b8582767b75eb405226f11c2: Status 404 returned error can't find the container with id 40cbb6d647b2a5b23bfddfe6b44f919c48018b47b8582767b75eb405226f11c2
Apr 16 16:55:19.957908 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:19.957875    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" event={"ID":"fc5f7cc7-9d96-4368-af29-e2c53b22bbd8","Type":"ContainerStarted","Data":"8ae0eb29659c90756c3256b5444e12589f267a5263064fc07f9b9dc0ba23d2dd"}
Apr 16 16:55:19.958290 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:19.957913    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" event={"ID":"fc5f7cc7-9d96-4368-af29-e2c53b22bbd8","Type":"ContainerStarted","Data":"40cbb6d647b2a5b23bfddfe6b44f919c48018b47b8582767b75eb405226f11c2"}
Apr 16 16:55:21.964938 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:21.964903    2578 generic.go:358] "Generic (PLEG): container finished" podID="319081fc-5b36-494b-82d5-2021e1f9b528" containerID="6706108dbd936be315704399f4e487053ec7a3ea57b1eb925f6640074f8a0fd2" exitCode=0
Apr 16 16:55:21.965311 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:21.964978    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq" event={"ID":"319081fc-5b36-494b-82d5-2021e1f9b528","Type":"ContainerDied","Data":"6706108dbd936be315704399f4e487053ec7a3ea57b1eb925f6640074f8a0fd2"}
Apr 16 16:55:22.209636 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:22.209615    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq"
Apr 16 16:55:22.294232 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:22.294165    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/319081fc-5b36-494b-82d5-2021e1f9b528-kserve-provision-location\") pod \"319081fc-5b36-494b-82d5-2021e1f9b528\" (UID: \"319081fc-5b36-494b-82d5-2021e1f9b528\") "
Apr 16 16:55:22.294447 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:22.294425    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319081fc-5b36-494b-82d5-2021e1f9b528-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "319081fc-5b36-494b-82d5-2021e1f9b528" (UID: "319081fc-5b36-494b-82d5-2021e1f9b528"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:55:22.395149 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:22.395119    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/319081fc-5b36-494b-82d5-2021e1f9b528-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:55:22.969824 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:22.969801    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq"
Apr 16 16:55:22.970226 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:22.969827    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq" event={"ID":"319081fc-5b36-494b-82d5-2021e1f9b528","Type":"ContainerDied","Data":"8077ba5e46caf004da67e521a3c2f0e1ba581c6321784822d408507dccc96c9c"}
Apr 16 16:55:22.970226 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:22.969872    2578 scope.go:117] "RemoveContainer" containerID="6706108dbd936be315704399f4e487053ec7a3ea57b1eb925f6640074f8a0fd2"
Apr 16 16:55:22.977924 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:22.977541    2578 scope.go:117] "RemoveContainer" containerID="7b82e0e1c4a50ef911ebb1db48b39b4fee73296ee2bd23ba09e65c8caeac3ccb"
Apr 16 16:55:22.990345 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:22.990325    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq"]
Apr 16 16:55:22.993645 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:22.993623    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-9l9zq"]
Apr 16 16:55:23.235050 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:23.234995    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="319081fc-5b36-494b-82d5-2021e1f9b528" path="/var/lib/kubelet/pods/319081fc-5b36-494b-82d5-2021e1f9b528/volumes"
Apr 16 16:55:23.974901 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:23.974875    2578 generic.go:358] "Generic (PLEG): container finished" podID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerID="8ae0eb29659c90756c3256b5444e12589f267a5263064fc07f9b9dc0ba23d2dd" exitCode=0
Apr 16 16:55:23.975232 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:23.974916    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" event={"ID":"fc5f7cc7-9d96-4368-af29-e2c53b22bbd8","Type":"ContainerDied","Data":"8ae0eb29659c90756c3256b5444e12589f267a5263064fc07f9b9dc0ba23d2dd"}
Apr 16 16:55:24.980343 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:24.980308    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" event={"ID":"fc5f7cc7-9d96-4368-af29-e2c53b22bbd8","Type":"ContainerStarted","Data":"ce8e9df171b2643b3b7617a04513ecb97253c1084d83e2bf5cb251da29a42350"}
Apr 16 16:55:24.980725 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:24.980587    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7"
Apr 16 16:55:24.981502 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:24.981480    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Apr 16 16:55:24.995879 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:24.995842    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" podStartSLOduration=6.995831786 podStartE2EDuration="6.995831786s" podCreationTimestamp="2026-04-16 16:55:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:55:24.995399051 +0000 UTC m=+1534.392615438" watchObservedRunningTime="2026-04-16 16:55:24.995831786 +0000 UTC m=+1534.393048174"
Apr 16 16:55:25.983875 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:25.983834    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Apr 16 16:55:35.984128 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:35.984086    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Apr 16 16:55:45.984010 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:45.983968    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Apr 16 16:55:55.984246 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:55:55.984205    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Apr 16 16:56:05.984067 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:05.984028    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Apr 16 16:56:15.984513 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:15.984473    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Apr 16 16:56:25.984599 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:25.984540    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Apr 16 16:56:35.984754 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:35.984663    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Apr 16 16:56:41.235078 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:41.235049    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7"
Apr 16 16:56:49.999858 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:49.999822    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7"]
Apr 16 16:56:50.000346 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:50.000157    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerName="kserve-container" containerID="cri-o://ce8e9df171b2643b3b7617a04513ecb97253c1084d83e2bf5cb251da29a42350" gracePeriod=30
Apr 16 16:56:50.059724 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:50.059695    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq"]
Apr 16 16:56:50.060005 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:50.059993    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="319081fc-5b36-494b-82d5-2021e1f9b528" containerName="storage-initializer"
Apr 16 16:56:50.060049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:50.060007    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="319081fc-5b36-494b-82d5-2021e1f9b528" containerName="storage-initializer"
Apr 16 16:56:50.060049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:50.060023    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="319081fc-5b36-494b-82d5-2021e1f9b528" containerName="kserve-container"
Apr 16 16:56:50.060049 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:50.060029    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="319081fc-5b36-494b-82d5-2021e1f9b528" containerName="kserve-container"
Apr 16 16:56:50.060141 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:50.060081    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="319081fc-5b36-494b-82d5-2021e1f9b528" containerName="kserve-container"
Apr 16 16:56:50.063018 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:50.063003    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq"
Apr 16 16:56:50.069856 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:50.069755    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq"]
Apr 16 16:56:50.102666 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:50.102634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd960f5e-ec3b-4548-b086-17deac4f27a7-kserve-provision-location\") pod \"isvc-primary-13dd4f-predictor-549cf5d885-wlpmq\" (UID: \"fd960f5e-ec3b-4548-b086-17deac4f27a7\") " pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq"
Apr 16 16:56:50.203643 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:50.203604    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd960f5e-ec3b-4548-b086-17deac4f27a7-kserve-provision-location\") pod \"isvc-primary-13dd4f-predictor-549cf5d885-wlpmq\" (UID: \"fd960f5e-ec3b-4548-b086-17deac4f27a7\") " pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq"
Apr 16 16:56:50.203973 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:50.203949    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd960f5e-ec3b-4548-b086-17deac4f27a7-kserve-provision-location\") pod \"isvc-primary-13dd4f-predictor-549cf5d885-wlpmq\" (UID: \"fd960f5e-ec3b-4548-b086-17deac4f27a7\") " pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq"
Apr 16 16:56:50.373948 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:50.373861    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq"
Apr 16 16:56:50.489084 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:50.489049    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq"]
Apr 16 16:56:50.492150 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:56:50.492125    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd960f5e_ec3b_4548_b086_17deac4f27a7.slice/crio-2ce967b16632d98fb4b3b9bed15939b9cc46f21c18d1015d67cff2ba584fa4e8 WatchSource:0}: Error finding container 2ce967b16632d98fb4b3b9bed15939b9cc46f21c18d1015d67cff2ba584fa4e8: Status 404 returned error can't find the container with id 2ce967b16632d98fb4b3b9bed15939b9cc46f21c18d1015d67cff2ba584fa4e8
Apr 16 16:56:51.233664 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:51.233626    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Apr 16 16:56:51.234705 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:51.234682    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq" event={"ID":"fd960f5e-ec3b-4548-b086-17deac4f27a7","Type":"ContainerStarted","Data":"35e7421adb1e7f103bf1f6825271e2da1cc717477da7ade787536ef14c453fc5"}
Apr 16 16:56:51.234830 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:51.234715    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq" event={"ID":"fd960f5e-ec3b-4548-b086-17deac4f27a7","Type":"ContainerStarted","Data":"2ce967b16632d98fb4b3b9bed15939b9cc46f21c18d1015d67cff2ba584fa4e8"}
Apr 16 16:56:53.131876 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:53.131852    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7"
Apr 16 16:56:53.223382 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:53.223362    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc5f7cc7-9d96-4368-af29-e2c53b22bbd8-kserve-provision-location\") pod \"fc5f7cc7-9d96-4368-af29-e2c53b22bbd8\" (UID: \"fc5f7cc7-9d96-4368-af29-e2c53b22bbd8\") "
Apr 16 16:56:53.223667 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:53.223646    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5f7cc7-9d96-4368-af29-e2c53b22bbd8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" (UID: "fc5f7cc7-9d96-4368-af29-e2c53b22bbd8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:56:53.236897 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:53.236869    2578 generic.go:358] "Generic (PLEG): container finished" podID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerID="ce8e9df171b2643b3b7617a04513ecb97253c1084d83e2bf5cb251da29a42350" exitCode=0
Apr 16 16:56:53.237005 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:53.236914    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" event={"ID":"fc5f7cc7-9d96-4368-af29-e2c53b22bbd8","Type":"ContainerDied","Data":"ce8e9df171b2643b3b7617a04513ecb97253c1084d83e2bf5cb251da29a42350"}
Apr 16 16:56:53.237005 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:53.236929    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7"
Apr 16 16:56:53.237005 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:53.236949    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7" event={"ID":"fc5f7cc7-9d96-4368-af29-e2c53b22bbd8","Type":"ContainerDied","Data":"40cbb6d647b2a5b23bfddfe6b44f919c48018b47b8582767b75eb405226f11c2"}
Apr 16 16:56:53.237005 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:53.236965    2578 scope.go:117] "RemoveContainer" containerID="ce8e9df171b2643b3b7617a04513ecb97253c1084d83e2bf5cb251da29a42350"
Apr 16 16:56:53.245968 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:53.245951    2578 scope.go:117] "RemoveContainer" containerID="8ae0eb29659c90756c3256b5444e12589f267a5263064fc07f9b9dc0ba23d2dd"
Apr 16 16:56:53.252516 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:53.252501    2578 scope.go:117] "RemoveContainer" containerID="ce8e9df171b2643b3b7617a04513ecb97253c1084d83e2bf5cb251da29a42350"
Apr 16 16:56:53.252753 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:56:53.252732    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8e9df171b2643b3b7617a04513ecb97253c1084d83e2bf5cb251da29a42350\": container with ID starting with ce8e9df171b2643b3b7617a04513ecb97253c1084d83e2bf5cb251da29a42350 not found: ID does not exist" containerID="ce8e9df171b2643b3b7617a04513ecb97253c1084d83e2bf5cb251da29a42350"
Apr 16 16:56:53.252812 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:53.252760    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8e9df171b2643b3b7617a04513ecb97253c1084d83e2bf5cb251da29a42350"} err="failed to get container status \"ce8e9df171b2643b3b7617a04513ecb97253c1084d83e2bf5cb251da29a42350\": rpc error: code = NotFound desc = could not find container \"ce8e9df171b2643b3b7617a04513ecb97253c1084d83e2bf5cb251da29a42350\": container with ID starting with ce8e9df171b2643b3b7617a04513ecb97253c1084d83e2bf5cb251da29a42350 not found: ID does not exist"
Apr 16 16:56:53.252812 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:53.252777    2578 scope.go:117] "RemoveContainer" containerID="8ae0eb29659c90756c3256b5444e12589f267a5263064fc07f9b9dc0ba23d2dd"
Apr 16 16:56:53.253007 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:56:53.252990    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae0eb29659c90756c3256b5444e12589f267a5263064fc07f9b9dc0ba23d2dd\": container with ID starting with 8ae0eb29659c90756c3256b5444e12589f267a5263064fc07f9b9dc0ba23d2dd not found: ID does not exist" containerID="8ae0eb29659c90756c3256b5444e12589f267a5263064fc07f9b9dc0ba23d2dd"
Apr 16 16:56:53.253064 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:53.253016    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae0eb29659c90756c3256b5444e12589f267a5263064fc07f9b9dc0ba23d2dd"} err="failed to get container status \"8ae0eb29659c90756c3256b5444e12589f267a5263064fc07f9b9dc0ba23d2dd\": rpc error: code = NotFound desc = could not find container \"8ae0eb29659c90756c3256b5444e12589f267a5263064fc07f9b9dc0ba23d2dd\": container with ID starting with 8ae0eb29659c90756c3256b5444e12589f267a5263064fc07f9b9dc0ba23d2dd not found: ID does not exist"
Apr 16 16:56:53.258110 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:53.258087    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7"]
Apr 16 16:56:53.263194 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:53.263175    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-59gc7"]
Apr 16 16:56:53.324243 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:53.324221    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc5f7cc7-9d96-4368-af29-e2c53b22bbd8-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:56:55.234567 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:55.234522    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" path="/var/lib/kubelet/pods/fc5f7cc7-9d96-4368-af29-e2c53b22bbd8/volumes"
Apr 16 16:56:55.244229 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:55.244205    2578 generic.go:358] "Generic (PLEG): container finished" podID="fd960f5e-ec3b-4548-b086-17deac4f27a7" containerID="35e7421adb1e7f103bf1f6825271e2da1cc717477da7ade787536ef14c453fc5" exitCode=0
Apr 16 16:56:55.244355 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:55.244256    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq" event={"ID":"fd960f5e-ec3b-4548-b086-17deac4f27a7","Type":"ContainerDied","Data":"35e7421adb1e7f103bf1f6825271e2da1cc717477da7ade787536ef14c453fc5"}
Apr 16 16:56:56.248386 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:56.248352    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq" event={"ID":"fd960f5e-ec3b-4548-b086-17deac4f27a7","Type":"ContainerStarted","Data":"f6b194f645759b68b67e4f6e2436ef96d990b5f71ce6c4812d0eb3f5d29839c3"}
Apr 16 16:56:56.248816 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:56.248644    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq"
Apr 16 16:56:56.249714 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:56.249687    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq" podUID="fd960f5e-ec3b-4548-b086-17deac4f27a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Apr 16 16:56:56.264820 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:56.264781    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq" podStartSLOduration=6.264771256 podStartE2EDuration="6.264771256s" podCreationTimestamp="2026-04-16 16:56:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:56:56.263757969 +0000 UTC m=+1625.660974358" watchObservedRunningTime="2026-04-16 16:56:56.264771256 +0000 UTC m=+1625.661987714"
Apr 16 16:56:57.251184 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:56:57.251150    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq" podUID="fd960f5e-ec3b-4548-b086-17deac4f27a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Apr 16 16:57:07.252171 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:57:07.252131    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq" podUID="fd960f5e-ec3b-4548-b086-17deac4f27a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Apr 16 16:57:17.251461 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:57:17.251419    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq" podUID="fd960f5e-ec3b-4548-b086-17deac4f27a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Apr 16 16:57:27.251390 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:57:27.251348    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq" podUID="fd960f5e-ec3b-4548-b086-17deac4f27a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Apr 16 16:57:37.251996 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:57:37.251958    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq" podUID="fd960f5e-ec3b-4548-b086-17deac4f27a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Apr 16 16:57:47.251412 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:57:47.251369    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq" podUID="fd960f5e-ec3b-4548-b086-17deac4f27a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Apr 16 16:57:57.251710 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:57:57.251683    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq"
Apr 16 16:58:00.208066 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.207988    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb"]
Apr 16 16:58:00.208410 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.208299    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerName="kserve-container"
Apr 16 16:58:00.208410 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.208309    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerName="kserve-container"
Apr 16 16:58:00.208410 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.208321    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerName="storage-initializer"
Apr 16 16:58:00.208410 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.208327    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerName="storage-initializer"
Apr 16 16:58:00.208410 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.208379    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="fc5f7cc7-9d96-4368-af29-e2c53b22bbd8" containerName="kserve-container"
Apr 16 16:58:00.211855 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.211839    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb"
Apr 16 16:58:00.213980 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.213952    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-13dd4f\""
Apr 16 16:58:00.214128 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.214032    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-13dd4f-dockercfg-flgk7\""
Apr 16 16:58:00.214812 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.214796    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Apr 16 16:58:00.218114 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.218090    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb"]
Apr 16 16:58:00.378731 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.378693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1169ebc-532c-4f20-856c-f84600476ef3-kserve-provision-location\") pod \"isvc-secondary-13dd4f-predictor-f579df46d-jvclb\" (UID: \"a1169ebc-532c-4f20-856c-f84600476ef3\") " pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb"
Apr 16 16:58:00.378902 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.378759    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1169ebc-532c-4f20-856c-f84600476ef3-cabundle-cert\") pod \"isvc-secondary-13dd4f-predictor-f579df46d-jvclb\" (UID: \"a1169ebc-532c-4f20-856c-f84600476ef3\") " pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb"
Apr 16 16:58:00.479653 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.479577    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1169ebc-532c-4f20-856c-f84600476ef3-kserve-provision-location\") pod \"isvc-secondary-13dd4f-predictor-f579df46d-jvclb\" (UID: \"a1169ebc-532c-4f20-856c-f84600476ef3\") " pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb"
Apr 16 16:58:00.479653 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.479638    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1169ebc-532c-4f20-856c-f84600476ef3-cabundle-cert\") pod \"isvc-secondary-13dd4f-predictor-f579df46d-jvclb\" (UID: \"a1169ebc-532c-4f20-856c-f84600476ef3\") " pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb"
Apr 16 16:58:00.479896 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.479880    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1169ebc-532c-4f20-856c-f84600476ef3-kserve-provision-location\") pod \"isvc-secondary-13dd4f-predictor-f579df46d-jvclb\" (UID: \"a1169ebc-532c-4f20-856c-f84600476ef3\") " pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb"
Apr 16 16:58:00.480164 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.480148    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1169ebc-532c-4f20-856c-f84600476ef3-cabundle-cert\") pod \"isvc-secondary-13dd4f-predictor-f579df46d-jvclb\" (UID: \"a1169ebc-532c-4f20-856c-f84600476ef3\") " pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb"
Apr 16 16:58:00.522813 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.522795    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb"
Apr 16 16:58:00.638600 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.638568    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb"]
Apr 16 16:58:00.641961 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:58:00.641933    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1169ebc_532c_4f20_856c_f84600476ef3.slice/crio-110c28c6707b0bb0293078f5790fd41ff24d4999862177707c290c0edc1bde1c WatchSource:0}: Error finding container 110c28c6707b0bb0293078f5790fd41ff24d4999862177707c290c0edc1bde1c: Status 404 returned error can't find the container with id 110c28c6707b0bb0293078f5790fd41ff24d4999862177707c290c0edc1bde1c
Apr 16 16:58:00.643660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:00.643642    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 16:58:01.437617 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:01.437570    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb" event={"ID":"a1169ebc-532c-4f20-856c-f84600476ef3","Type":"ContainerStarted","Data":"5e0e1b34f9ef5f0e998eb2b68554e3d4996ea918d0473b516953f9bea48975c9"}
Apr 16 16:58:01.437617 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:01.437613    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb" event={"ID":"a1169ebc-532c-4f20-856c-f84600476ef3","Type":"ContainerStarted","Data":"110c28c6707b0bb0293078f5790fd41ff24d4999862177707c290c0edc1bde1c"}
Apr 16 16:58:04.448992 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:04.448961    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-13dd4f-predictor-f579df46d-jvclb_a1169ebc-532c-4f20-856c-f84600476ef3/storage-initializer/0.log"
Apr 16 16:58:04.449437 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:04.449001    2578 generic.go:358] "Generic (PLEG): container finished" podID="a1169ebc-532c-4f20-856c-f84600476ef3" containerID="5e0e1b34f9ef5f0e998eb2b68554e3d4996ea918d0473b516953f9bea48975c9" exitCode=1
Apr 16 16:58:04.449437 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:04.449027    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb" event={"ID":"a1169ebc-532c-4f20-856c-f84600476ef3","Type":"ContainerDied","Data":"5e0e1b34f9ef5f0e998eb2b68554e3d4996ea918d0473b516953f9bea48975c9"}
Apr 16 16:58:05.453971 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:05.453940    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-13dd4f-predictor-f579df46d-jvclb_a1169ebc-532c-4f20-856c-f84600476ef3/storage-initializer/0.log"
Apr 16 16:58:05.454309 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:05.454063    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb" event={"ID":"a1169ebc-532c-4f20-856c-f84600476ef3","Type":"ContainerStarted","Data":"7eb764f46511a8eeaf504b262175d081c4fc64a26b111942f317eb0535d81c70"}
Apr 16 16:58:10.470264 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:10.470237    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-13dd4f-predictor-f579df46d-jvclb_a1169ebc-532c-4f20-856c-f84600476ef3/storage-initializer/1.log"
Apr 16 16:58:10.470719 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:10.470632    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-13dd4f-predictor-f579df46d-jvclb_a1169ebc-532c-4f20-856c-f84600476ef3/storage-initializer/0.log"
Apr 16 16:58:10.470719 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:10.470673    2578 generic.go:358] "Generic (PLEG): container finished" podID="a1169ebc-532c-4f20-856c-f84600476ef3" containerID="7eb764f46511a8eeaf504b262175d081c4fc64a26b111942f317eb0535d81c70" exitCode=1
Apr 16 16:58:10.470833 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:10.470723    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb" event={"ID":"a1169ebc-532c-4f20-856c-f84600476ef3","Type":"ContainerDied","Data":"7eb764f46511a8eeaf504b262175d081c4fc64a26b111942f317eb0535d81c70"}
Apr 16 16:58:10.470833 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:10.470765    2578 scope.go:117] "RemoveContainer" containerID="5e0e1b34f9ef5f0e998eb2b68554e3d4996ea918d0473b516953f9bea48975c9"
Apr 16 16:58:10.471122 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:10.471102    2578 scope.go:117] "RemoveContainer" containerID="5e0e1b34f9ef5f0e998eb2b68554e3d4996ea918d0473b516953f9bea48975c9"
Apr 16 16:58:10.480899 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:58:10.480871    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-13dd4f-predictor-f579df46d-jvclb_kserve-ci-e2e-test_a1169ebc-532c-4f20-856c-f84600476ef3_0 in pod sandbox 110c28c6707b0bb0293078f5790fd41ff24d4999862177707c290c0edc1bde1c from index: no such id: '5e0e1b34f9ef5f0e998eb2b68554e3d4996ea918d0473b516953f9bea48975c9'" containerID="5e0e1b34f9ef5f0e998eb2b68554e3d4996ea918d0473b516953f9bea48975c9"
Apr 16 16:58:10.480977 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:58:10.480919    2578 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-13dd4f-predictor-f579df46d-jvclb_kserve-ci-e2e-test_a1169ebc-532c-4f20-856c-f84600476ef3_0 in pod sandbox 110c28c6707b0bb0293078f5790fd41ff24d4999862177707c290c0edc1bde1c from index: no such id: '5e0e1b34f9ef5f0e998eb2b68554e3d4996ea918d0473b516953f9bea48975c9'; Skipping pod \"isvc-secondary-13dd4f-predictor-f579df46d-jvclb_kserve-ci-e2e-test(a1169ebc-532c-4f20-856c-f84600476ef3)\"" logger="UnhandledError"
Apr 16 16:58:10.482246 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:58:10.482226    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-13dd4f-predictor-f579df46d-jvclb_kserve-ci-e2e-test(a1169ebc-532c-4f20-856c-f84600476ef3)\"" pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb" podUID="a1169ebc-532c-4f20-856c-f84600476ef3"
Apr 16 16:58:11.474912 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:11.474886    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-13dd4f-predictor-f579df46d-jvclb_a1169ebc-532c-4f20-856c-f84600476ef3/storage-initializer/1.log"
Apr 16 16:58:16.298194 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.298165    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb"]
Apr 16 16:58:16.350333 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.350294    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq"]
Apr 16 16:58:16.351701 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.350648    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq" podUID="fd960f5e-ec3b-4548-b086-17deac4f27a7" containerName="kserve-container" containerID="cri-o://f6b194f645759b68b67e4f6e2436ef96d990b5f71ce6c4812d0eb3f5d29839c3" gracePeriod=30
Apr 16 16:58:16.398047 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.398021    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2"]
Apr 16 16:58:16.402703 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.402683    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2"
Apr 16 16:58:16.404998 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.404974    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-0a4b4d\""
Apr 16 16:58:16.405111 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.405015    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-0a4b4d-dockercfg-47vp6\""
Apr 16 16:58:16.409660 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.409638    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2"]
Apr 16 16:58:16.427525 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.427511    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-13dd4f-predictor-f579df46d-jvclb_a1169ebc-532c-4f20-856c-f84600476ef3/storage-initializer/1.log"
Apr 16 16:58:16.427623 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.427576    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb"
Apr 16 16:58:16.487253 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.487226    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1169ebc-532c-4f20-856c-f84600476ef3-cabundle-cert\") pod \"a1169ebc-532c-4f20-856c-f84600476ef3\" (UID: \"a1169ebc-532c-4f20-856c-f84600476ef3\") "
Apr 16 16:58:16.487373 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.487279    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1169ebc-532c-4f20-856c-f84600476ef3-kserve-provision-location\") pod \"a1169ebc-532c-4f20-856c-f84600476ef3\" (UID: \"a1169ebc-532c-4f20-856c-f84600476ef3\") "
Apr 16 16:58:16.487445 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.487383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01bac299-c6b3-428e-a744-19edb5ad19fa-kserve-provision-location\") pod \"isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2\" (UID: \"01bac299-c6b3-428e-a744-19edb5ad19fa\") " pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2"
Apr 16 16:58:16.487504 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.487472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01bac299-c6b3-428e-a744-19edb5ad19fa-cabundle-cert\") pod \"isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2\" (UID: \"01bac299-c6b3-428e-a744-19edb5ad19fa\") " pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2"
Apr 16 16:58:16.487591 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.487536    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1169ebc-532c-4f20-856c-f84600476ef3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a1169ebc-532c-4f20-856c-f84600476ef3" (UID: "a1169ebc-532c-4f20-856c-f84600476ef3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:58:16.487591 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.487544    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1169ebc-532c-4f20-856c-f84600476ef3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a1169ebc-532c-4f20-856c-f84600476ef3" (UID: "a1169ebc-532c-4f20-856c-f84600476ef3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:58:16.490040 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.490021    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-13dd4f-predictor-f579df46d-jvclb_a1169ebc-532c-4f20-856c-f84600476ef3/storage-initializer/1.log"
Apr 16 16:58:16.490131 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.490069    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb" event={"ID":"a1169ebc-532c-4f20-856c-f84600476ef3","Type":"ContainerDied","Data":"110c28c6707b0bb0293078f5790fd41ff24d4999862177707c290c0edc1bde1c"}
Apr 16 16:58:16.490131 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.490100    2578 scope.go:117] "RemoveContainer" containerID="7eb764f46511a8eeaf504b262175d081c4fc64a26b111942f317eb0535d81c70"
Apr 16 16:58:16.490131 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.490116    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb"
Apr 16 16:58:16.524715 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.524691    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb"]
Apr 16 16:58:16.526847 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.526828    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-13dd4f-predictor-f579df46d-jvclb"]
Apr 16 16:58:16.588228 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.588157    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01bac299-c6b3-428e-a744-19edb5ad19fa-cabundle-cert\") pod \"isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2\" (UID: \"01bac299-c6b3-428e-a744-19edb5ad19fa\") " pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2"
Apr 16 16:58:16.588228 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.588200    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01bac299-c6b3-428e-a744-19edb5ad19fa-kserve-provision-location\") pod \"isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2\" (UID: \"01bac299-c6b3-428e-a744-19edb5ad19fa\") " pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2"
Apr 16 16:58:16.588388 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.588252    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1169ebc-532c-4f20-856c-f84600476ef3-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:58:16.588388 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.588268    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1169ebc-532c-4f20-856c-f84600476ef3-cabundle-cert\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:58:16.588596 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.588580    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01bac299-c6b3-428e-a744-19edb5ad19fa-kserve-provision-location\") pod \"isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2\" (UID: \"01bac299-c6b3-428e-a744-19edb5ad19fa\") " pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2"
Apr 16 16:58:16.588742 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.588725    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01bac299-c6b3-428e-a744-19edb5ad19fa-cabundle-cert\") pod \"isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2\" (UID: \"01bac299-c6b3-428e-a744-19edb5ad19fa\") " pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2"
Apr 16 16:58:16.726130 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.726112    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2"
Apr 16 16:58:16.845061 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:16.845039    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2"]
Apr 16 16:58:16.847222 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:58:16.847194    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01bac299_c6b3_428e_a744_19edb5ad19fa.slice/crio-d9a7bee4753d9787a16fa249bdbc5c73880f02dc7fcc274c78b91caf178a268e WatchSource:0}: Error finding container d9a7bee4753d9787a16fa249bdbc5c73880f02dc7fcc274c78b91caf178a268e: Status 404 returned error can't find the container with id d9a7bee4753d9787a16fa249bdbc5c73880f02dc7fcc274c78b91caf178a268e
Apr 16 16:58:17.236368 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:17.236336    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1169ebc-532c-4f20-856c-f84600476ef3" path="/var/lib/kubelet/pods/a1169ebc-532c-4f20-856c-f84600476ef3/volumes"
Apr 16 16:58:17.251453 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:17.251424    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq" podUID="fd960f5e-ec3b-4548-b086-17deac4f27a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Apr 16 16:58:17.494824 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:17.494732    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2" event={"ID":"01bac299-c6b3-428e-a744-19edb5ad19fa","Type":"ContainerStarted","Data":"f5c05e26b8aee8261d330f0a987e25dd47a4f081cf3db2ca642a93d6a483bea4"}
Apr 16 16:58:17.494824 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:17.494771    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2" event={"ID":"01bac299-c6b3-428e-a744-19edb5ad19fa","Type":"ContainerStarted","Data":"d9a7bee4753d9787a16fa249bdbc5c73880f02dc7fcc274c78b91caf178a268e"}
Apr 16 16:58:20.176772 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:20.176750    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq"
Apr 16 16:58:20.212729 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:20.212703    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd960f5e-ec3b-4548-b086-17deac4f27a7-kserve-provision-location\") pod \"fd960f5e-ec3b-4548-b086-17deac4f27a7\" (UID: \"fd960f5e-ec3b-4548-b086-17deac4f27a7\") "
Apr 16 16:58:20.213005 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:20.212985    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd960f5e-ec3b-4548-b086-17deac4f27a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd960f5e-ec3b-4548-b086-17deac4f27a7" (UID: "fd960f5e-ec3b-4548-b086-17deac4f27a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:58:20.313856 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:20.313805    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd960f5e-ec3b-4548-b086-17deac4f27a7-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:58:20.507409 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:20.507380    2578 generic.go:358] "Generic (PLEG): container finished" podID="fd960f5e-ec3b-4548-b086-17deac4f27a7" containerID="f6b194f645759b68b67e4f6e2436ef96d990b5f71ce6c4812d0eb3f5d29839c3" exitCode=0
Apr 16 16:58:20.507571 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:20.507458    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq" event={"ID":"fd960f5e-ec3b-4548-b086-17deac4f27a7","Type":"ContainerDied","Data":"f6b194f645759b68b67e4f6e2436ef96d990b5f71ce6c4812d0eb3f5d29839c3"}
Apr 16 16:58:20.507571 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:20.507482    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq" event={"ID":"fd960f5e-ec3b-4548-b086-17deac4f27a7","Type":"ContainerDied","Data":"2ce967b16632d98fb4b3b9bed15939b9cc46f21c18d1015d67cff2ba584fa4e8"}
Apr 16 16:58:20.507571 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:20.507496    2578 scope.go:117] "RemoveContainer" containerID="f6b194f645759b68b67e4f6e2436ef96d990b5f71ce6c4812d0eb3f5d29839c3"
Apr 16 16:58:20.507571 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:20.507518    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq"
Apr 16 16:58:20.515658 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:20.515453    2578 scope.go:117] "RemoveContainer" containerID="35e7421adb1e7f103bf1f6825271e2da1cc717477da7ade787536ef14c453fc5"
Apr 16 16:58:20.522503 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:20.522488    2578 scope.go:117] "RemoveContainer" containerID="f6b194f645759b68b67e4f6e2436ef96d990b5f71ce6c4812d0eb3f5d29839c3"
Apr 16 16:58:20.522736 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:58:20.522718    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b194f645759b68b67e4f6e2436ef96d990b5f71ce6c4812d0eb3f5d29839c3\": container with ID starting with f6b194f645759b68b67e4f6e2436ef96d990b5f71ce6c4812d0eb3f5d29839c3 not found: ID does not exist" containerID="f6b194f645759b68b67e4f6e2436ef96d990b5f71ce6c4812d0eb3f5d29839c3"
Apr 16 16:58:20.522784 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:20.522745    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b194f645759b68b67e4f6e2436ef96d990b5f71ce6c4812d0eb3f5d29839c3"} err="failed to get container status \"f6b194f645759b68b67e4f6e2436ef96d990b5f71ce6c4812d0eb3f5d29839c3\": rpc error: code = NotFound desc = could not find container \"f6b194f645759b68b67e4f6e2436ef96d990b5f71ce6c4812d0eb3f5d29839c3\": container with ID starting with f6b194f645759b68b67e4f6e2436ef96d990b5f71ce6c4812d0eb3f5d29839c3 not found: ID does not exist"
Apr 16 16:58:20.522784 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:20.522762    2578 scope.go:117] "RemoveContainer" containerID="35e7421adb1e7f103bf1f6825271e2da1cc717477da7ade787536ef14c453fc5"
Apr 16 16:58:20.522995 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:58:20.522979    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e7421adb1e7f103bf1f6825271e2da1cc717477da7ade787536ef14c453fc5\": container with ID starting with 35e7421adb1e7f103bf1f6825271e2da1cc717477da7ade787536ef14c453fc5 not found: ID does not exist" containerID="35e7421adb1e7f103bf1f6825271e2da1cc717477da7ade787536ef14c453fc5"
Apr 16 16:58:20.523045 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:20.522998    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e7421adb1e7f103bf1f6825271e2da1cc717477da7ade787536ef14c453fc5"} err="failed to get container status \"35e7421adb1e7f103bf1f6825271e2da1cc717477da7ade787536ef14c453fc5\": rpc error: code = NotFound desc = could not find container \"35e7421adb1e7f103bf1f6825271e2da1cc717477da7ade787536ef14c453fc5\": container with ID starting with 35e7421adb1e7f103bf1f6825271e2da1cc717477da7ade787536ef14c453fc5 not found: ID does not exist"
Apr 16 16:58:20.527925 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:20.527889    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq"]
Apr 16 16:58:20.529294 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:20.529271    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-13dd4f-predictor-549cf5d885-wlpmq"]
Apr 16 16:58:21.234650 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:21.234613    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd960f5e-ec3b-4548-b086-17deac4f27a7" path="/var/lib/kubelet/pods/fd960f5e-ec3b-4548-b086-17deac4f27a7/volumes"
Apr 16 16:58:23.517431 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:23.517406    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2_01bac299-c6b3-428e-a744-19edb5ad19fa/storage-initializer/0.log"
Apr 16 16:58:23.517779 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:23.517440    2578 generic.go:358] "Generic (PLEG): container finished" podID="01bac299-c6b3-428e-a744-19edb5ad19fa" containerID="f5c05e26b8aee8261d330f0a987e25dd47a4f081cf3db2ca642a93d6a483bea4" exitCode=1
Apr 16 16:58:23.517779 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:23.517478    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2" event={"ID":"01bac299-c6b3-428e-a744-19edb5ad19fa","Type":"ContainerDied","Data":"f5c05e26b8aee8261d330f0a987e25dd47a4f081cf3db2ca642a93d6a483bea4"}
Apr 16 16:58:24.522096 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:24.522068    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2_01bac299-c6b3-428e-a744-19edb5ad19fa/storage-initializer/0.log"
Apr 16 16:58:24.522435 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:24.522143    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2" event={"ID":"01bac299-c6b3-428e-a744-19edb5ad19fa","Type":"ContainerStarted","Data":"10a604c8e684db7261a119e6a487ca43b914f1a49beaf7b393fa687c300f7490"}
Apr 16 16:58:26.426750 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.426720    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2"]
Apr 16 16:58:26.427132 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.426945    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2" podUID="01bac299-c6b3-428e-a744-19edb5ad19fa" containerName="storage-initializer" containerID="cri-o://10a604c8e684db7261a119e6a487ca43b914f1a49beaf7b393fa687c300f7490" gracePeriod=30
Apr 16 16:58:26.543525 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.543495    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg"]
Apr 16 16:58:26.543831 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.543819    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd960f5e-ec3b-4548-b086-17deac4f27a7" containerName="storage-initializer"
Apr 16 16:58:26.543876 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.543833    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd960f5e-ec3b-4548-b086-17deac4f27a7" containerName="storage-initializer"
Apr 16 16:58:26.543876 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.543843    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd960f5e-ec3b-4548-b086-17deac4f27a7" containerName="kserve-container"
Apr 16 16:58:26.543876 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.543849    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd960f5e-ec3b-4548-b086-17deac4f27a7" containerName="kserve-container"
Apr 16 16:58:26.543876 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.543861    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1169ebc-532c-4f20-856c-f84600476ef3" containerName="storage-initializer"
Apr 16 16:58:26.543876 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.543867    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1169ebc-532c-4f20-856c-f84600476ef3" containerName="storage-initializer"
Apr 16 16:58:26.544070 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.543882    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1169ebc-532c-4f20-856c-f84600476ef3" containerName="storage-initializer"
Apr 16 16:58:26.544070 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.543887    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1169ebc-532c-4f20-856c-f84600476ef3" containerName="storage-initializer"
Apr 16 16:58:26.544070 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.543932    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a1169ebc-532c-4f20-856c-f84600476ef3" containerName="storage-initializer"
Apr 16 16:58:26.544070 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.543944    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="fd960f5e-ec3b-4548-b086-17deac4f27a7" containerName="kserve-container"
Apr 16 16:58:26.544070 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.544035    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a1169ebc-532c-4f20-856c-f84600476ef3" containerName="storage-initializer"
Apr 16 16:58:26.546923 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.546897    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg"
Apr 16 16:58:26.548984 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.548964    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fhlr7\""
Apr 16 16:58:26.557293 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.557267    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg"]
Apr 16 16:58:26.560080 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.560061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b64067-91d7-4254-a5af-da73bdcc3adb-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-85bccb8945-8d5qg\" (UID: \"d0b64067-91d7-4254-a5af-da73bdcc3adb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg"
Apr 16 16:58:26.660358 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.660321    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b64067-91d7-4254-a5af-da73bdcc3adb-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-85bccb8945-8d5qg\" (UID: \"d0b64067-91d7-4254-a5af-da73bdcc3adb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg"
Apr 16 16:58:26.660650 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.660630    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b64067-91d7-4254-a5af-da73bdcc3adb-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-85bccb8945-8d5qg\" (UID: \"d0b64067-91d7-4254-a5af-da73bdcc3adb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg"
Apr 16 16:58:26.856491 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.856463    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg"
Apr 16 16:58:26.970356 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:26.970278    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg"]
Apr 16 16:58:26.972875 ip-10-0-140-164 kubenswrapper[2578]: W0416 16:58:26.972847    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b64067_91d7_4254_a5af_da73bdcc3adb.slice/crio-1c9607e62332c46c03ed41d0f44da81a6888dc0047b1e1d02c29c97f0c5e582c WatchSource:0}: Error finding container 1c9607e62332c46c03ed41d0f44da81a6888dc0047b1e1d02c29c97f0c5e582c: Status 404 returned error can't find the container with id 1c9607e62332c46c03ed41d0f44da81a6888dc0047b1e1d02c29c97f0c5e582c
Apr 16 16:58:27.531998 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:27.531966    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" event={"ID":"d0b64067-91d7-4254-a5af-da73bdcc3adb","Type":"ContainerStarted","Data":"593c3d47acfe834f4fafe71939c4a634921f5db2b8c807c1ba96313c0e931ffb"}
Apr 16 16:58:27.532371 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:27.532006    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" event={"ID":"d0b64067-91d7-4254-a5af-da73bdcc3adb","Type":"ContainerStarted","Data":"1c9607e62332c46c03ed41d0f44da81a6888dc0047b1e1d02c29c97f0c5e582c"}
Apr 16 16:58:28.256823 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.256801    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2_01bac299-c6b3-428e-a744-19edb5ad19fa/storage-initializer/1.log"
Apr 16 16:58:28.257152 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.257139    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2_01bac299-c6b3-428e-a744-19edb5ad19fa/storage-initializer/0.log"
Apr 16 16:58:28.257219 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.257192    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2"
Apr 16 16:58:28.272861 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.272842    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01bac299-c6b3-428e-a744-19edb5ad19fa-kserve-provision-location\") pod \"01bac299-c6b3-428e-a744-19edb5ad19fa\" (UID: \"01bac299-c6b3-428e-a744-19edb5ad19fa\") "
Apr 16 16:58:28.272939 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.272882    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01bac299-c6b3-428e-a744-19edb5ad19fa-cabundle-cert\") pod \"01bac299-c6b3-428e-a744-19edb5ad19fa\" (UID: \"01bac299-c6b3-428e-a744-19edb5ad19fa\") "
Apr 16 16:58:28.273114 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.273080    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01bac299-c6b3-428e-a744-19edb5ad19fa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "01bac299-c6b3-428e-a744-19edb5ad19fa" (UID: "01bac299-c6b3-428e-a744-19edb5ad19fa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:58:28.273264 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.273243    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01bac299-c6b3-428e-a744-19edb5ad19fa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "01bac299-c6b3-428e-a744-19edb5ad19fa" (UID: "01bac299-c6b3-428e-a744-19edb5ad19fa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:58:28.273348 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.273302    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01bac299-c6b3-428e-a744-19edb5ad19fa-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:58:28.373935 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.373872    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01bac299-c6b3-428e-a744-19edb5ad19fa-cabundle-cert\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 16:58:28.535850 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.535830    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2_01bac299-c6b3-428e-a744-19edb5ad19fa/storage-initializer/1.log"
Apr 16 16:58:28.536198 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.536176    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2_01bac299-c6b3-428e-a744-19edb5ad19fa/storage-initializer/0.log"
Apr 16 16:58:28.536242 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.536205    2578 generic.go:358] "Generic (PLEG): container finished" podID="01bac299-c6b3-428e-a744-19edb5ad19fa" containerID="10a604c8e684db7261a119e6a487ca43b914f1a49beaf7b393fa687c300f7490" exitCode=1
Apr 16 16:58:28.536277 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.536233    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2" event={"ID":"01bac299-c6b3-428e-a744-19edb5ad19fa","Type":"ContainerDied","Data":"10a604c8e684db7261a119e6a487ca43b914f1a49beaf7b393fa687c300f7490"}
Apr 16 16:58:28.536313 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.536275    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2" event={"ID":"01bac299-c6b3-428e-a744-19edb5ad19fa","Type":"ContainerDied","Data":"d9a7bee4753d9787a16fa249bdbc5c73880f02dc7fcc274c78b91caf178a268e"}
Apr 16 16:58:28.536313 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.536286    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2"
Apr 16 16:58:28.536313 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.536297    2578 scope.go:117] "RemoveContainer" containerID="10a604c8e684db7261a119e6a487ca43b914f1a49beaf7b393fa687c300f7490"
Apr 16 16:58:28.544480 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.544460    2578 scope.go:117] "RemoveContainer" containerID="f5c05e26b8aee8261d330f0a987e25dd47a4f081cf3db2ca642a93d6a483bea4"
Apr 16 16:58:28.551585 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.551568    2578 scope.go:117] "RemoveContainer" containerID="10a604c8e684db7261a119e6a487ca43b914f1a49beaf7b393fa687c300f7490"
Apr 16 16:58:28.551816 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:58:28.551799    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10a604c8e684db7261a119e6a487ca43b914f1a49beaf7b393fa687c300f7490\": container with ID starting with 10a604c8e684db7261a119e6a487ca43b914f1a49beaf7b393fa687c300f7490 not found: ID does not exist" containerID="10a604c8e684db7261a119e6a487ca43b914f1a49beaf7b393fa687c300f7490"
Apr 16 16:58:28.551889 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.551828    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a604c8e684db7261a119e6a487ca43b914f1a49beaf7b393fa687c300f7490"} err="failed to get container status \"10a604c8e684db7261a119e6a487ca43b914f1a49beaf7b393fa687c300f7490\": rpc error: code = NotFound desc = could not find container \"10a604c8e684db7261a119e6a487ca43b914f1a49beaf7b393fa687c300f7490\": container with ID starting with 10a604c8e684db7261a119e6a487ca43b914f1a49beaf7b393fa687c300f7490 not found: ID does not exist"
Apr 16 16:58:28.551889 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.551853    2578 scope.go:117] "RemoveContainer" containerID="f5c05e26b8aee8261d330f0a987e25dd47a4f081cf3db2ca642a93d6a483bea4"
Apr 16 16:58:28.552098 ip-10-0-140-164 kubenswrapper[2578]: E0416 16:58:28.552083    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c05e26b8aee8261d330f0a987e25dd47a4f081cf3db2ca642a93d6a483bea4\": container with ID starting with f5c05e26b8aee8261d330f0a987e25dd47a4f081cf3db2ca642a93d6a483bea4 not found: ID does not exist" containerID="f5c05e26b8aee8261d330f0a987e25dd47a4f081cf3db2ca642a93d6a483bea4"
Apr 16 16:58:28.552138 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.552104    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c05e26b8aee8261d330f0a987e25dd47a4f081cf3db2ca642a93d6a483bea4"} err="failed to get container status \"f5c05e26b8aee8261d330f0a987e25dd47a4f081cf3db2ca642a93d6a483bea4\": rpc error: code = NotFound desc = could not find container \"f5c05e26b8aee8261d330f0a987e25dd47a4f081cf3db2ca642a93d6a483bea4\": container with ID starting with f5c05e26b8aee8261d330f0a987e25dd47a4f081cf3db2ca642a93d6a483bea4 not found: ID does not exist"
Apr 16 16:58:28.569218 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.569190    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2"]
Apr 16 16:58:28.574585 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:28.574540    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0a4b4d-predictor-988d6c764-74cp2"]
Apr 16 16:58:29.235429 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:29.235395    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01bac299-c6b3-428e-a744-19edb5ad19fa" path="/var/lib/kubelet/pods/01bac299-c6b3-428e-a744-19edb5ad19fa/volumes"
Apr 16 16:58:31.546648 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:31.546611    2578 generic.go:358] "Generic (PLEG): container finished" podID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerID="593c3d47acfe834f4fafe71939c4a634921f5db2b8c807c1ba96313c0e931ffb" exitCode=0
Apr 16 16:58:31.547106 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:31.546683    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" event={"ID":"d0b64067-91d7-4254-a5af-da73bdcc3adb","Type":"ContainerDied","Data":"593c3d47acfe834f4fafe71939c4a634921f5db2b8c807c1ba96313c0e931ffb"}
Apr 16 16:58:48.603517 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:48.603485    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" event={"ID":"d0b64067-91d7-4254-a5af-da73bdcc3adb","Type":"ContainerStarted","Data":"6e7fd8ae138e20bac5107d0206d8ebfcd12d25a38ee726f8010e6142fb3df559"}
Apr 16 16:58:48.603978 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:48.603779    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg"
Apr 16 16:58:48.604852 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:48.604826    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" podUID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Apr 16 16:58:48.619261 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:48.619217    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" podStartSLOduration=6.36070479 podStartE2EDuration="22.61920561s" podCreationTimestamp="2026-04-16 16:58:26 +0000 UTC" firstStartedPulling="2026-04-16 16:58:31.547851023 +0000 UTC m=+1720.945067389" lastFinishedPulling="2026-04-16 16:58:47.806351842 +0000 UTC m=+1737.203568209" observedRunningTime="2026-04-16 16:58:48.617366912 +0000 UTC m=+1738.014583302" watchObservedRunningTime="2026-04-16 16:58:48.61920561 +0000 UTC m=+1738.016422000"
Apr 16 16:58:49.606417 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:49.606373    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" podUID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Apr 16 16:58:59.607279 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:58:59.607244    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" podUID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Apr 16 16:59:09.606474 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:59:09.606433    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" podUID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Apr 16 16:59:19.606362 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:59:19.606324    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" podUID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Apr 16 16:59:29.606423 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:59:29.606331    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" podUID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Apr 16 16:59:39.607121 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:59:39.607080    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" podUID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Apr 16 16:59:49.606592 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:59:49.606530    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" podUID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Apr 16 16:59:51.248372 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:59:51.248345    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 16:59:51.251648 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:59:51.251629    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 16:59:59.607395 ip-10-0-140-164 kubenswrapper[2578]: I0416 16:59:59.607363    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg"
Apr 16 17:00:06.705885 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:06.705854    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg"]
Apr 16 17:00:06.706438 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:06.706088    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" podUID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerName="kserve-container" containerID="cri-o://6e7fd8ae138e20bac5107d0206d8ebfcd12d25a38ee726f8010e6142fb3df559" gracePeriod=30
Apr 16 17:00:06.787536 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:06.787510    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4"]
Apr 16 17:00:06.787834 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:06.787813    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01bac299-c6b3-428e-a744-19edb5ad19fa" containerName="storage-initializer"
Apr 16 17:00:06.787834 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:06.787830    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="01bac299-c6b3-428e-a744-19edb5ad19fa" containerName="storage-initializer"
Apr 16 17:00:06.787981 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:06.787894    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="01bac299-c6b3-428e-a744-19edb5ad19fa" containerName="storage-initializer"
Apr 16 17:00:06.787981 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:06.787950    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01bac299-c6b3-428e-a744-19edb5ad19fa" containerName="storage-initializer"
Apr 16 17:00:06.787981 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:06.787957    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="01bac299-c6b3-428e-a744-19edb5ad19fa" containerName="storage-initializer"
Apr 16 17:00:06.788081 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:06.788013    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="01bac299-c6b3-428e-a744-19edb5ad19fa" containerName="storage-initializer"
Apr 16 17:00:06.790769 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:06.790753    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4"
Apr 16 17:00:06.798459 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:06.798437    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4"]
Apr 16 17:00:06.885212 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:06.885181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4\" (UID: \"2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4"
Apr 16 17:00:06.985781 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:06.985713    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4\" (UID: \"2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4"
Apr 16 17:00:06.986089 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:06.986069    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4\" (UID: \"2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4"
Apr 16 17:00:07.102071 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:07.102046    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4"
Apr 16 17:00:07.215166 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:07.215143    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4"]
Apr 16 17:00:07.217646 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:00:07.217618    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9ec6ee_4d14_4f03_b96a_4f369cf9ba39.slice/crio-694dd2c5e261cb3a48b6daaa26b5aae491a332c0fbccf7170c1c7d72f8c8c66c WatchSource:0}: Error finding container 694dd2c5e261cb3a48b6daaa26b5aae491a332c0fbccf7170c1c7d72f8c8c66c: Status 404 returned error can't find the container with id 694dd2c5e261cb3a48b6daaa26b5aae491a332c0fbccf7170c1c7d72f8c8c66c
Apr 16 17:00:07.834487 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:07.834446    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4" event={"ID":"2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39","Type":"ContainerStarted","Data":"8b445b703834bd0a067fa2a158077dd8579c35fc78d937dd8a259c449d2b18c6"}
Apr 16 17:00:07.834487 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:07.834492    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4" event={"ID":"2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39","Type":"ContainerStarted","Data":"694dd2c5e261cb3a48b6daaa26b5aae491a332c0fbccf7170c1c7d72f8c8c66c"}
Apr 16 17:00:09.607232 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:09.607190    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" podUID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Apr 16 17:00:10.836342 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:10.836322    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg"
Apr 16 17:00:10.845424 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:10.845397    2578 generic.go:358] "Generic (PLEG): container finished" podID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerID="6e7fd8ae138e20bac5107d0206d8ebfcd12d25a38ee726f8010e6142fb3df559" exitCode=0
Apr 16 17:00:10.845542 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:10.845451    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" event={"ID":"d0b64067-91d7-4254-a5af-da73bdcc3adb","Type":"ContainerDied","Data":"6e7fd8ae138e20bac5107d0206d8ebfcd12d25a38ee726f8010e6142fb3df559"}
Apr 16 17:00:10.845542 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:10.845488    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg" event={"ID":"d0b64067-91d7-4254-a5af-da73bdcc3adb","Type":"ContainerDied","Data":"1c9607e62332c46c03ed41d0f44da81a6888dc0047b1e1d02c29c97f0c5e582c"}
Apr 16 17:00:10.845542 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:10.845502    2578 scope.go:117] "RemoveContainer" containerID="6e7fd8ae138e20bac5107d0206d8ebfcd12d25a38ee726f8010e6142fb3df559"
Apr 16 17:00:10.845542 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:10.845459    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg"
Apr 16 17:00:10.853142 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:10.853105    2578 scope.go:117] "RemoveContainer" containerID="593c3d47acfe834f4fafe71939c4a634921f5db2b8c807c1ba96313c0e931ffb"
Apr 16 17:00:10.860646 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:10.860630    2578 scope.go:117] "RemoveContainer" containerID="6e7fd8ae138e20bac5107d0206d8ebfcd12d25a38ee726f8010e6142fb3df559"
Apr 16 17:00:10.860928 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:00:10.860911    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e7fd8ae138e20bac5107d0206d8ebfcd12d25a38ee726f8010e6142fb3df559\": container with ID starting with 6e7fd8ae138e20bac5107d0206d8ebfcd12d25a38ee726f8010e6142fb3df559 not found: ID does not exist" containerID="6e7fd8ae138e20bac5107d0206d8ebfcd12d25a38ee726f8010e6142fb3df559"
Apr 16 17:00:10.860992 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:10.860939    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e7fd8ae138e20bac5107d0206d8ebfcd12d25a38ee726f8010e6142fb3df559"} err="failed to get container status \"6e7fd8ae138e20bac5107d0206d8ebfcd12d25a38ee726f8010e6142fb3df559\": rpc error: code = NotFound desc = could not find container \"6e7fd8ae138e20bac5107d0206d8ebfcd12d25a38ee726f8010e6142fb3df559\": container with ID starting with 6e7fd8ae138e20bac5107d0206d8ebfcd12d25a38ee726f8010e6142fb3df559 not found: ID does not exist"
Apr 16 17:00:10.860992 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:10.860961    2578 scope.go:117] "RemoveContainer" containerID="593c3d47acfe834f4fafe71939c4a634921f5db2b8c807c1ba96313c0e931ffb"
Apr 16 17:00:10.861246 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:00:10.861220    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"593c3d47acfe834f4fafe71939c4a634921f5db2b8c807c1ba96313c0e931ffb\": container with ID starting with 593c3d47acfe834f4fafe71939c4a634921f5db2b8c807c1ba96313c0e931ffb not found: ID does not exist" containerID="593c3d47acfe834f4fafe71939c4a634921f5db2b8c807c1ba96313c0e931ffb"
Apr 16 17:00:10.861306 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:10.861250    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"593c3d47acfe834f4fafe71939c4a634921f5db2b8c807c1ba96313c0e931ffb"} err="failed to get container status \"593c3d47acfe834f4fafe71939c4a634921f5db2b8c807c1ba96313c0e931ffb\": rpc error: code = NotFound desc = could not find container \"593c3d47acfe834f4fafe71939c4a634921f5db2b8c807c1ba96313c0e931ffb\": container with ID starting with 593c3d47acfe834f4fafe71939c4a634921f5db2b8c807c1ba96313c0e931ffb not found: ID does not exist"
Apr 16 17:00:10.917334 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:10.917289    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b64067-91d7-4254-a5af-da73bdcc3adb-kserve-provision-location\") pod \"d0b64067-91d7-4254-a5af-da73bdcc3adb\" (UID: \"d0b64067-91d7-4254-a5af-da73bdcc3adb\") "
Apr 16 17:00:10.917570 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:10.917532    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b64067-91d7-4254-a5af-da73bdcc3adb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0b64067-91d7-4254-a5af-da73bdcc3adb" (UID: "d0b64067-91d7-4254-a5af-da73bdcc3adb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:00:11.018164 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:11.018140    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b64067-91d7-4254-a5af-da73bdcc3adb-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:00:11.166046 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:11.166023    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg"]
Apr 16 17:00:11.167408 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:11.167366    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-8d5qg"]
Apr 16 17:00:11.234449 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:11.234427    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0b64067-91d7-4254-a5af-da73bdcc3adb" path="/var/lib/kubelet/pods/d0b64067-91d7-4254-a5af-da73bdcc3adb/volumes"
Apr 16 17:00:11.850393 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:11.850367    2578 generic.go:358] "Generic (PLEG): container finished" podID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" containerID="8b445b703834bd0a067fa2a158077dd8579c35fc78d937dd8a259c449d2b18c6" exitCode=0
Apr 16 17:00:11.850760 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:11.850436    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4" event={"ID":"2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39","Type":"ContainerDied","Data":"8b445b703834bd0a067fa2a158077dd8579c35fc78d937dd8a259c449d2b18c6"}
Apr 16 17:00:12.855062 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:12.855026    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4" event={"ID":"2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39","Type":"ContainerStarted","Data":"f403f9daca0141c4302bfc33642814a7db095e2cbd6eb4fa9f516f2b66902e20"}
Apr 16 17:00:12.855531 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:12.855357    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4"
Apr 16 17:00:12.856716 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:12.856690    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4" podUID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Apr 16 17:00:12.870452 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:12.870416    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4" podStartSLOduration=6.87040568 podStartE2EDuration="6.87040568s" podCreationTimestamp="2026-04-16 17:00:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:00:12.868863043 +0000 UTC m=+1822.266079431" watchObservedRunningTime="2026-04-16 17:00:12.87040568 +0000 UTC m=+1822.267622068"
Apr 16 17:00:13.858362 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:13.858321    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4" podUID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Apr 16 17:00:23.858892 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:23.858846    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4" podUID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Apr 16 17:00:33.859092 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:33.859050    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4" podUID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Apr 16 17:00:43.858852 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:43.858809    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4" podUID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Apr 16 17:00:53.858681 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:00:53.858637    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4" podUID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Apr 16 17:01:03.859015 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:03.858974    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4" podUID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Apr 16 17:01:13.858896 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:13.858855    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4" podUID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Apr 16 17:01:23.235224 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:23.235193    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4"
Apr 16 17:01:26.904840 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:26.904804    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4"]
Apr 16 17:01:26.905293 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:26.905141    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4" podUID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" containerName="kserve-container" containerID="cri-o://f403f9daca0141c4302bfc33642814a7db095e2cbd6eb4fa9f516f2b66902e20" gracePeriod=30
Apr 16 17:01:26.953536 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:26.953508    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx"]
Apr 16 17:01:26.953842 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:26.953828    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerName="storage-initializer"
Apr 16 17:01:26.953842 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:26.953843    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerName="storage-initializer"
Apr 16 17:01:26.953947 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:26.953861    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerName="kserve-container"
Apr 16 17:01:26.953947 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:26.953867    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerName="kserve-container"
Apr 16 17:01:26.953947 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:26.953922    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="d0b64067-91d7-4254-a5af-da73bdcc3adb" containerName="kserve-container"
Apr 16 17:01:26.956749 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:26.956735    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx"
Apr 16 17:01:26.964625 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:26.964600    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx"]
Apr 16 17:01:27.045886 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:27.045855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e014ea96-b310-46ee-bf5d-bc81c9ef54e2-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-669896799c-c5lgx\" (UID: \"e014ea96-b310-46ee-bf5d-bc81c9ef54e2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx"
Apr 16 17:01:27.146160 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:27.146138    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e014ea96-b310-46ee-bf5d-bc81c9ef54e2-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-669896799c-c5lgx\" (UID: \"e014ea96-b310-46ee-bf5d-bc81c9ef54e2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx"
Apr 16 17:01:27.146471 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:27.146452    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e014ea96-b310-46ee-bf5d-bc81c9ef54e2-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-669896799c-c5lgx\" (UID: \"e014ea96-b310-46ee-bf5d-bc81c9ef54e2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx"
Apr 16 17:01:27.267406 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:27.267382    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx"
Apr 16 17:01:27.382006 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:27.381980    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx"]
Apr 16 17:01:27.384282 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:01:27.384243    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode014ea96_b310_46ee_bf5d_bc81c9ef54e2.slice/crio-697a9f18e32dadc1c57a36dfb2c4a8c74c01309e91ff998880c853d5f14ce713 WatchSource:0}: Error finding container 697a9f18e32dadc1c57a36dfb2c4a8c74c01309e91ff998880c853d5f14ce713: Status 404 returned error can't find the container with id 697a9f18e32dadc1c57a36dfb2c4a8c74c01309e91ff998880c853d5f14ce713
Apr 16 17:01:28.065835 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:28.065796    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx" event={"ID":"e014ea96-b310-46ee-bf5d-bc81c9ef54e2","Type":"ContainerStarted","Data":"0e6379d6bb0ee2505095ea3f88075ed33db525feabd82dffacd6191a57b37b4d"}
Apr 16 17:01:28.066207 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:28.065843    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx" event={"ID":"e014ea96-b310-46ee-bf5d-bc81c9ef54e2","Type":"ContainerStarted","Data":"697a9f18e32dadc1c57a36dfb2c4a8c74c01309e91ff998880c853d5f14ce713"}
Apr 16 17:01:30.840787 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:30.840768    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4"
Apr 16 17:01:30.874278 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:30.874255    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39-kserve-provision-location\") pod \"2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39\" (UID: \"2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39\") "
Apr 16 17:01:30.874532 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:30.874512    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" (UID: "2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:01:30.975138 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:30.975105    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:01:31.074666 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:31.074640    2578 generic.go:358] "Generic (PLEG): container finished" podID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" containerID="f403f9daca0141c4302bfc33642814a7db095e2cbd6eb4fa9f516f2b66902e20" exitCode=0
Apr 16 17:01:31.074773 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:31.074700    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4"
Apr 16 17:01:31.074773 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:31.074730    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4" event={"ID":"2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39","Type":"ContainerDied","Data":"f403f9daca0141c4302bfc33642814a7db095e2cbd6eb4fa9f516f2b66902e20"}
Apr 16 17:01:31.074894 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:31.074773    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4" event={"ID":"2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39","Type":"ContainerDied","Data":"694dd2c5e261cb3a48b6daaa26b5aae491a332c0fbccf7170c1c7d72f8c8c66c"}
Apr 16 17:01:31.074894 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:31.074794    2578 scope.go:117] "RemoveContainer" containerID="f403f9daca0141c4302bfc33642814a7db095e2cbd6eb4fa9f516f2b66902e20"
Apr 16 17:01:31.083267 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:31.083249    2578 scope.go:117] "RemoveContainer" containerID="8b445b703834bd0a067fa2a158077dd8579c35fc78d937dd8a259c449d2b18c6"
Apr 16 17:01:31.090045 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:31.090030    2578 scope.go:117] "RemoveContainer" containerID="f403f9daca0141c4302bfc33642814a7db095e2cbd6eb4fa9f516f2b66902e20"
Apr 16 17:01:31.090270 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:01:31.090253    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f403f9daca0141c4302bfc33642814a7db095e2cbd6eb4fa9f516f2b66902e20\": container with ID starting with f403f9daca0141c4302bfc33642814a7db095e2cbd6eb4fa9f516f2b66902e20 not found: ID does not exist" containerID="f403f9daca0141c4302bfc33642814a7db095e2cbd6eb4fa9f516f2b66902e20"
Apr 16 17:01:31.090315 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:31.090278    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f403f9daca0141c4302bfc33642814a7db095e2cbd6eb4fa9f516f2b66902e20"} err="failed to get container status \"f403f9daca0141c4302bfc33642814a7db095e2cbd6eb4fa9f516f2b66902e20\": rpc error: code = NotFound desc = could not find container \"f403f9daca0141c4302bfc33642814a7db095e2cbd6eb4fa9f516f2b66902e20\": container with ID starting with f403f9daca0141c4302bfc33642814a7db095e2cbd6eb4fa9f516f2b66902e20 not found: ID does not exist"
Apr 16 17:01:31.090315 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:31.090294    2578 scope.go:117] "RemoveContainer" containerID="8b445b703834bd0a067fa2a158077dd8579c35fc78d937dd8a259c449d2b18c6"
Apr 16 17:01:31.090523 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:01:31.090505    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b445b703834bd0a067fa2a158077dd8579c35fc78d937dd8a259c449d2b18c6\": container with ID starting with 8b445b703834bd0a067fa2a158077dd8579c35fc78d937dd8a259c449d2b18c6 not found: ID does not exist" containerID="8b445b703834bd0a067fa2a158077dd8579c35fc78d937dd8a259c449d2b18c6"
Apr 16 17:01:31.090591 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:31.090528    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b445b703834bd0a067fa2a158077dd8579c35fc78d937dd8a259c449d2b18c6"} err="failed to get container status \"8b445b703834bd0a067fa2a158077dd8579c35fc78d937dd8a259c449d2b18c6\": rpc error: code = NotFound desc = could not find container \"8b445b703834bd0a067fa2a158077dd8579c35fc78d937dd8a259c449d2b18c6\": container with ID starting with 8b445b703834bd0a067fa2a158077dd8579c35fc78d937dd8a259c449d2b18c6 not found: ID does not exist"
Apr 16 17:01:31.098365 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:31.098342    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4"]
Apr 16 17:01:31.100529 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:31.100510    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-9nzq4"]
Apr 16 17:01:31.235118 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:31.235061    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" path="/var/lib/kubelet/pods/2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39/volumes"
Apr 16 17:01:32.079384 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:32.079356    2578 generic.go:358] "Generic (PLEG): container finished" podID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" containerID="0e6379d6bb0ee2505095ea3f88075ed33db525feabd82dffacd6191a57b37b4d" exitCode=0
Apr 16 17:01:32.079840 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:32.079421    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx" event={"ID":"e014ea96-b310-46ee-bf5d-bc81c9ef54e2","Type":"ContainerDied","Data":"0e6379d6bb0ee2505095ea3f88075ed33db525feabd82dffacd6191a57b37b4d"}
Apr 16 17:01:33.083579 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:33.083537    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx" event={"ID":"e014ea96-b310-46ee-bf5d-bc81c9ef54e2","Type":"ContainerStarted","Data":"88c1d2dd01647ad51c3f811c35a3cd347954acb09323cdeb3cb8524d8d52379b"}
Apr 16 17:01:33.084019 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:33.083878    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx"
Apr 16 17:01:33.085100 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:33.085073    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx" podUID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Apr 16 17:01:33.103668 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:33.103624    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx" podStartSLOduration=7.103614162 podStartE2EDuration="7.103614162s" podCreationTimestamp="2026-04-16 17:01:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:01:33.102439651 +0000 UTC m=+1902.499656040" watchObservedRunningTime="2026-04-16 17:01:33.103614162 +0000 UTC m=+1902.500830550"
Apr 16 17:01:34.091323 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:34.091267    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx" podUID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Apr 16 17:01:44.088249 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:44.088207    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx" podUID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Apr 16 17:01:54.088280 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:01:54.088239    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx" podUID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Apr 16 17:02:04.088133 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:04.088092    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx" podUID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Apr 16 17:02:14.088089 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:14.088049    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx" podUID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Apr 16 17:02:24.088097 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:24.088049    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx" podUID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Apr 16 17:02:34.088413 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:34.088328    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx" podUID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Apr 16 17:02:44.089173 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:44.089144    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx"
Apr 16 17:02:47.110256 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:47.110224    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx"]
Apr 16 17:02:47.110658 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:47.110466    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx" podUID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" containerName="kserve-container" containerID="cri-o://88c1d2dd01647ad51c3f811c35a3cd347954acb09323cdeb3cb8524d8d52379b" gracePeriod=30
Apr 16 17:02:47.169112 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:47.169084    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb"]
Apr 16 17:02:47.169434 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:47.169421    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" containerName="kserve-container"
Apr 16 17:02:47.169478 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:47.169436    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" containerName="kserve-container"
Apr 16 17:02:47.169478 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:47.169453    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" containerName="storage-initializer"
Apr 16 17:02:47.169478 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:47.169459    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" containerName="storage-initializer"
Apr 16 17:02:47.169603 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:47.169529    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="2d9ec6ee-4d14-4f03-b96a-4f369cf9ba39" containerName="kserve-container"
Apr 16 17:02:47.172648 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:47.172632    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb"
Apr 16 17:02:47.181028 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:47.181004    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb"]
Apr 16 17:02:47.227105 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:47.227078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd9ce055-b7c9-4361-8416-f10e54402b64-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb\" (UID: \"fd9ce055-b7c9-4361-8416-f10e54402b64\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb"
Apr 16 17:02:47.327534 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:47.327503    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd9ce055-b7c9-4361-8416-f10e54402b64-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb\" (UID: \"fd9ce055-b7c9-4361-8416-f10e54402b64\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb"
Apr 16 17:02:47.327873 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:47.327850    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd9ce055-b7c9-4361-8416-f10e54402b64-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb\" (UID: \"fd9ce055-b7c9-4361-8416-f10e54402b64\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb"
Apr 16 17:02:47.482702 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:47.482674    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb"
Apr 16 17:02:47.601039 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:47.601010    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb"]
Apr 16 17:02:47.603652 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:02:47.603624    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9ce055_b7c9_4361_8416_f10e54402b64.slice/crio-8a45f0dd3e59c02839de6089ce8141018db3920550384fb91c83e7b237cf6849 WatchSource:0}: Error finding container 8a45f0dd3e59c02839de6089ce8141018db3920550384fb91c83e7b237cf6849: Status 404 returned error can't find the container with id 8a45f0dd3e59c02839de6089ce8141018db3920550384fb91c83e7b237cf6849
Apr 16 17:02:48.299839 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:48.299804    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb" event={"ID":"fd9ce055-b7c9-4361-8416-f10e54402b64","Type":"ContainerStarted","Data":"396734c0d71cb3493d4126e7516d9fad4217e232bec43b28bea57f2b8fe4d026"}
Apr 16 17:02:48.299839 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:48.299838    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb" event={"ID":"fd9ce055-b7c9-4361-8416-f10e54402b64","Type":"ContainerStarted","Data":"8a45f0dd3e59c02839de6089ce8141018db3920550384fb91c83e7b237cf6849"}
Apr 16 17:02:51.308875 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:51.308846    2578 generic.go:358] "Generic (PLEG): container finished" podID="fd9ce055-b7c9-4361-8416-f10e54402b64" containerID="396734c0d71cb3493d4126e7516d9fad4217e232bec43b28bea57f2b8fe4d026" exitCode=0
Apr 16 17:02:51.309201 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:51.308882    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb" event={"ID":"fd9ce055-b7c9-4361-8416-f10e54402b64","Type":"ContainerDied","Data":"396734c0d71cb3493d4126e7516d9fad4217e232bec43b28bea57f2b8fe4d026"}
Apr 16 17:02:51.946219 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:51.946194    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx"
Apr 16 17:02:51.964292 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:51.964069    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e014ea96-b310-46ee-bf5d-bc81c9ef54e2-kserve-provision-location\") pod \"e014ea96-b310-46ee-bf5d-bc81c9ef54e2\" (UID: \"e014ea96-b310-46ee-bf5d-bc81c9ef54e2\") "
Apr 16 17:02:51.964507 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:51.964476    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e014ea96-b310-46ee-bf5d-bc81c9ef54e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e014ea96-b310-46ee-bf5d-bc81c9ef54e2" (UID: "e014ea96-b310-46ee-bf5d-bc81c9ef54e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:02:52.065654 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:52.065574    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e014ea96-b310-46ee-bf5d-bc81c9ef54e2-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:02:52.314354 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:52.314318    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb" event={"ID":"fd9ce055-b7c9-4361-8416-f10e54402b64","Type":"ContainerStarted","Data":"89adc6b835cfae9de3ebc85c34fd654e7154b6367131507cbdc7eab12b432e62"}
Apr 16 17:02:52.314744 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:52.314599    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb"
Apr 16 17:02:52.315754 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:52.315694    2578 generic.go:358] "Generic (PLEG): container finished" podID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" containerID="88c1d2dd01647ad51c3f811c35a3cd347954acb09323cdeb3cb8524d8d52379b" exitCode=0
Apr 16 17:02:52.315754 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:52.315724    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx" event={"ID":"e014ea96-b310-46ee-bf5d-bc81c9ef54e2","Type":"ContainerDied","Data":"88c1d2dd01647ad51c3f811c35a3cd347954acb09323cdeb3cb8524d8d52379b"}
Apr 16 17:02:52.315754 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:52.315746    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx" event={"ID":"e014ea96-b310-46ee-bf5d-bc81c9ef54e2","Type":"ContainerDied","Data":"697a9f18e32dadc1c57a36dfb2c4a8c74c01309e91ff998880c853d5f14ce713"}
Apr 16 17:02:52.315950 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:52.315760    2578 scope.go:117] "RemoveContainer" containerID="88c1d2dd01647ad51c3f811c35a3cd347954acb09323cdeb3cb8524d8d52379b"
Apr 16 17:02:52.315950 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:52.315760    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx"
Apr 16 17:02:52.323948 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:52.323749    2578 scope.go:117] "RemoveContainer" containerID="0e6379d6bb0ee2505095ea3f88075ed33db525feabd82dffacd6191a57b37b4d"
Apr 16 17:02:52.330546 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:52.330530    2578 scope.go:117] "RemoveContainer" containerID="88c1d2dd01647ad51c3f811c35a3cd347954acb09323cdeb3cb8524d8d52379b"
Apr 16 17:02:52.330949 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:02:52.330923    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c1d2dd01647ad51c3f811c35a3cd347954acb09323cdeb3cb8524d8d52379b\": container with ID starting with 88c1d2dd01647ad51c3f811c35a3cd347954acb09323cdeb3cb8524d8d52379b not found: ID does not exist" containerID="88c1d2dd01647ad51c3f811c35a3cd347954acb09323cdeb3cb8524d8d52379b"
Apr 16 17:02:52.331085 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:52.330959    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c1d2dd01647ad51c3f811c35a3cd347954acb09323cdeb3cb8524d8d52379b"} err="failed to get container status \"88c1d2dd01647ad51c3f811c35a3cd347954acb09323cdeb3cb8524d8d52379b\": rpc error: code = NotFound desc = could not find container \"88c1d2dd01647ad51c3f811c35a3cd347954acb09323cdeb3cb8524d8d52379b\": container with ID starting with 88c1d2dd01647ad51c3f811c35a3cd347954acb09323cdeb3cb8524d8d52379b not found: ID does not exist"
Apr 16 17:02:52.331085 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:52.330983    2578 scope.go:117] "RemoveContainer" containerID="0e6379d6bb0ee2505095ea3f88075ed33db525feabd82dffacd6191a57b37b4d"
Apr 16 17:02:52.331322 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:02:52.331296    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e6379d6bb0ee2505095ea3f88075ed33db525feabd82dffacd6191a57b37b4d\": container with ID starting with 0e6379d6bb0ee2505095ea3f88075ed33db525feabd82dffacd6191a57b37b4d not found: ID does not exist" containerID="0e6379d6bb0ee2505095ea3f88075ed33db525feabd82dffacd6191a57b37b4d"
Apr 16 17:02:52.331419 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:52.331328    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e6379d6bb0ee2505095ea3f88075ed33db525feabd82dffacd6191a57b37b4d"} err="failed to get container status \"0e6379d6bb0ee2505095ea3f88075ed33db525feabd82dffacd6191a57b37b4d\": rpc error: code = NotFound desc = could not find container \"0e6379d6bb0ee2505095ea3f88075ed33db525feabd82dffacd6191a57b37b4d\": container with ID starting with 0e6379d6bb0ee2505095ea3f88075ed33db525feabd82dffacd6191a57b37b4d not found: ID does not exist"
Apr 16 17:02:52.332807 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:52.332774    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb" podStartSLOduration=5.332763783 podStartE2EDuration="5.332763783s" podCreationTimestamp="2026-04-16 17:02:47 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:02:52.330753151 +0000 UTC m=+1981.727969540" watchObservedRunningTime="2026-04-16 17:02:52.332763783 +0000 UTC m=+1981.729980171"
Apr 16 17:02:52.343404 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:52.343384    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx"]
Apr 16 17:02:52.351970 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:52.351952    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-c5lgx"]
Apr 16 17:02:53.235424 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:02:53.235395    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" path="/var/lib/kubelet/pods/e014ea96-b310-46ee-bf5d-bc81c9ef54e2/volumes"
Apr 16 17:03:23.321795 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:03:23.321756    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb" podUID="fd9ce055-b7c9-4361-8416-f10e54402b64" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.48:8080: connect: connection refused"
Apr 16 17:03:33.320665 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:03:33.320619    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb" podUID="fd9ce055-b7c9-4361-8416-f10e54402b64" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.48:8080: connect: connection refused"
Apr 16 17:03:43.320927 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:03:43.320876    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb" podUID="fd9ce055-b7c9-4361-8416-f10e54402b64" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.48:8080: connect: connection refused"
Apr 16 17:03:53.321328 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:03:53.321291    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb" podUID="fd9ce055-b7c9-4361-8416-f10e54402b64" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.48:8080: connect: connection refused"
Apr 16 17:04:03.324270 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:03.324188    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb"
Apr 16 17:04:07.302857 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:07.302828    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb"]
Apr 16 17:04:07.303224 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:07.303048    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb" podUID="fd9ce055-b7c9-4361-8416-f10e54402b64" containerName="kserve-container" containerID="cri-o://89adc6b835cfae9de3ebc85c34fd654e7154b6367131507cbdc7eab12b432e62" gracePeriod=30
Apr 16 17:04:07.354361 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:07.354332    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l"]
Apr 16 17:04:07.354879 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:07.354862    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" containerName="storage-initializer"
Apr 16 17:04:07.354879 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:07.354881    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" containerName="storage-initializer"
Apr 16 17:04:07.354995 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:07.354894    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" containerName="kserve-container"
Apr 16 17:04:07.354995 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:07.354899    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" containerName="kserve-container"
Apr 16 17:04:07.354995 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:07.354945    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="e014ea96-b310-46ee-bf5d-bc81c9ef54e2" containerName="kserve-container"
Apr 16 17:04:07.357882 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:07.357860    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l"
Apr 16 17:04:07.364821 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:07.364795    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l"]
Apr 16 17:04:07.393847 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:07.393819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de7978af-cbd9-4d2a-9ef1-33bc535c37f2-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l\" (UID: \"de7978af-cbd9-4d2a-9ef1-33bc535c37f2\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l"
Apr 16 17:04:07.494161 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:07.494129    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de7978af-cbd9-4d2a-9ef1-33bc535c37f2-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l\" (UID: \"de7978af-cbd9-4d2a-9ef1-33bc535c37f2\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l"
Apr 16 17:04:07.494467 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:07.494451    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de7978af-cbd9-4d2a-9ef1-33bc535c37f2-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l\" (UID: \"de7978af-cbd9-4d2a-9ef1-33bc535c37f2\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l"
Apr 16 17:04:07.669430 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:07.669370    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l"
Apr 16 17:04:07.783971 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:07.783948    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l"]
Apr 16 17:04:07.786140 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:04:07.786108    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7978af_cbd9_4d2a_9ef1_33bc535c37f2.slice/crio-714f059be6bedec199fa9857e57533f82f2d7c670487564489e3efdcdfedf403 WatchSource:0}: Error finding container 714f059be6bedec199fa9857e57533f82f2d7c670487564489e3efdcdfedf403: Status 404 returned error can't find the container with id 714f059be6bedec199fa9857e57533f82f2d7c670487564489e3efdcdfedf403
Apr 16 17:04:07.788065 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:07.788048    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 17:04:08.537915 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:08.537885    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l" event={"ID":"de7978af-cbd9-4d2a-9ef1-33bc535c37f2","Type":"ContainerStarted","Data":"c9d1dca1aa6cec31cf71e007cd9004ecfce06bdbfad72c5ec8c3ebab79dcd3b4"}
Apr 16 17:04:08.537915 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:08.537919    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l" event={"ID":"de7978af-cbd9-4d2a-9ef1-33bc535c37f2","Type":"ContainerStarted","Data":"714f059be6bedec199fa9857e57533f82f2d7c670487564489e3efdcdfedf403"}
Apr 16 17:04:11.337977 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:11.337953    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb"
Apr 16 17:04:11.523029 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:11.522997    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd9ce055-b7c9-4361-8416-f10e54402b64-kserve-provision-location\") pod \"fd9ce055-b7c9-4361-8416-f10e54402b64\" (UID: \"fd9ce055-b7c9-4361-8416-f10e54402b64\") "
Apr 16 17:04:11.523279 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:11.523255    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9ce055-b7c9-4361-8416-f10e54402b64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd9ce055-b7c9-4361-8416-f10e54402b64" (UID: "fd9ce055-b7c9-4361-8416-f10e54402b64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:04:11.547412 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:11.547389    2578 generic.go:358] "Generic (PLEG): container finished" podID="fd9ce055-b7c9-4361-8416-f10e54402b64" containerID="89adc6b835cfae9de3ebc85c34fd654e7154b6367131507cbdc7eab12b432e62" exitCode=0
Apr 16 17:04:11.547525 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:11.547451    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb"
Apr 16 17:04:11.547525 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:11.547468    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb" event={"ID":"fd9ce055-b7c9-4361-8416-f10e54402b64","Type":"ContainerDied","Data":"89adc6b835cfae9de3ebc85c34fd654e7154b6367131507cbdc7eab12b432e62"}
Apr 16 17:04:11.547525 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:11.547506    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb" event={"ID":"fd9ce055-b7c9-4361-8416-f10e54402b64","Type":"ContainerDied","Data":"8a45f0dd3e59c02839de6089ce8141018db3920550384fb91c83e7b237cf6849"}
Apr 16 17:04:11.547525 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:11.547521    2578 scope.go:117] "RemoveContainer" containerID="89adc6b835cfae9de3ebc85c34fd654e7154b6367131507cbdc7eab12b432e62"
Apr 16 17:04:11.555540 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:11.555503    2578 scope.go:117] "RemoveContainer" containerID="396734c0d71cb3493d4126e7516d9fad4217e232bec43b28bea57f2b8fe4d026"
Apr 16 17:04:11.564502 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:11.564488    2578 scope.go:117] "RemoveContainer" containerID="89adc6b835cfae9de3ebc85c34fd654e7154b6367131507cbdc7eab12b432e62"
Apr 16 17:04:11.564819 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:04:11.564794    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89adc6b835cfae9de3ebc85c34fd654e7154b6367131507cbdc7eab12b432e62\": container with ID starting with 89adc6b835cfae9de3ebc85c34fd654e7154b6367131507cbdc7eab12b432e62 not found: ID does not exist" containerID="89adc6b835cfae9de3ebc85c34fd654e7154b6367131507cbdc7eab12b432e62"
Apr 16 17:04:11.564897 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:11.564831    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89adc6b835cfae9de3ebc85c34fd654e7154b6367131507cbdc7eab12b432e62"} err="failed to get container status \"89adc6b835cfae9de3ebc85c34fd654e7154b6367131507cbdc7eab12b432e62\": rpc error: code = NotFound desc = could not find container \"89adc6b835cfae9de3ebc85c34fd654e7154b6367131507cbdc7eab12b432e62\": container with ID starting with 89adc6b835cfae9de3ebc85c34fd654e7154b6367131507cbdc7eab12b432e62 not found: ID does not exist"
Apr 16 17:04:11.564897 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:11.564856    2578 scope.go:117] "RemoveContainer" containerID="396734c0d71cb3493d4126e7516d9fad4217e232bec43b28bea57f2b8fe4d026"
Apr 16 17:04:11.565071 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:04:11.565050    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396734c0d71cb3493d4126e7516d9fad4217e232bec43b28bea57f2b8fe4d026\": container with ID starting with 396734c0d71cb3493d4126e7516d9fad4217e232bec43b28bea57f2b8fe4d026 not found: ID does not exist" containerID="396734c0d71cb3493d4126e7516d9fad4217e232bec43b28bea57f2b8fe4d026"
Apr 16 17:04:11.565134 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:11.565076    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396734c0d71cb3493d4126e7516d9fad4217e232bec43b28bea57f2b8fe4d026"} err="failed to get container status \"396734c0d71cb3493d4126e7516d9fad4217e232bec43b28bea57f2b8fe4d026\": rpc error: code = NotFound desc = could not find container \"396734c0d71cb3493d4126e7516d9fad4217e232bec43b28bea57f2b8fe4d026\": container with ID starting with 396734c0d71cb3493d4126e7516d9fad4217e232bec43b28bea57f2b8fe4d026 not found: ID does not exist"
Apr 16 17:04:11.567984 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:11.567964    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb"]
Apr 16 17:04:11.571349 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:11.571331    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-rsvtb"]
Apr 16 17:04:11.624355 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:11.624336    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd9ce055-b7c9-4361-8416-f10e54402b64-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:04:12.553073 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:12.553042    2578 generic.go:358] "Generic (PLEG): container finished" podID="de7978af-cbd9-4d2a-9ef1-33bc535c37f2" containerID="c9d1dca1aa6cec31cf71e007cd9004ecfce06bdbfad72c5ec8c3ebab79dcd3b4" exitCode=0
Apr 16 17:04:12.553429 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:12.553118    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l" event={"ID":"de7978af-cbd9-4d2a-9ef1-33bc535c37f2","Type":"ContainerDied","Data":"c9d1dca1aa6cec31cf71e007cd9004ecfce06bdbfad72c5ec8c3ebab79dcd3b4"}
Apr 16 17:04:13.236519 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:13.236488    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd9ce055-b7c9-4361-8416-f10e54402b64" path="/var/lib/kubelet/pods/fd9ce055-b7c9-4361-8416-f10e54402b64/volumes"
Apr 16 17:04:13.557807 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:13.557733    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l" event={"ID":"de7978af-cbd9-4d2a-9ef1-33bc535c37f2","Type":"ContainerStarted","Data":"807de508d0804cef24dfcac5f7a6a6d4eb07f158112cb3472ea49d33e9d29e7d"}
Apr 16 17:04:13.558127 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:13.557959    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l"
Apr 16 17:04:13.572334 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:13.572296    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l" podStartSLOduration=6.5722843399999995 podStartE2EDuration="6.57228434s" podCreationTimestamp="2026-04-16 17:04:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:04:13.571354217 +0000 UTC m=+2062.968570607" watchObservedRunningTime="2026-04-16 17:04:13.57228434 +0000 UTC m=+2062.969500728"
Apr 16 17:04:44.562488 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:44.562446    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l" podUID="de7978af-cbd9-4d2a-9ef1-33bc535c37f2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.49:8080: connect: connection refused"
Apr 16 17:04:51.269155 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:51.269125    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 17:04:51.273954 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:51.273936    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 17:04:54.561732 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:04:54.561690    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l" podUID="de7978af-cbd9-4d2a-9ef1-33bc535c37f2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.49:8080: connect: connection refused"
Apr 16 17:05:04.562205 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:04.562165    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l" podUID="de7978af-cbd9-4d2a-9ef1-33bc535c37f2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.49:8080: connect: connection refused"
Apr 16 17:05:14.562279 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:14.562234    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l" podUID="de7978af-cbd9-4d2a-9ef1-33bc535c37f2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.49:8080: connect: connection refused"
Apr 16 17:05:24.564900 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:24.564867    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l"
Apr 16 17:05:27.476898 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:27.476866    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l"]
Apr 16 17:05:27.477384 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:27.477204    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l" podUID="de7978af-cbd9-4d2a-9ef1-33bc535c37f2" containerName="kserve-container" containerID="cri-o://807de508d0804cef24dfcac5f7a6a6d4eb07f158112cb3472ea49d33e9d29e7d" gracePeriod=30
Apr 16 17:05:27.545656 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:27.545627    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft"]
Apr 16 17:05:27.545955 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:27.545942    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd9ce055-b7c9-4361-8416-f10e54402b64" containerName="storage-initializer"
Apr 16 17:05:27.546001 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:27.545956    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd9ce055-b7c9-4361-8416-f10e54402b64" containerName="storage-initializer"
Apr 16 17:05:27.546001 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:27.545969    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd9ce055-b7c9-4361-8416-f10e54402b64" containerName="kserve-container"
Apr 16 17:05:27.546001 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:27.545974    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd9ce055-b7c9-4361-8416-f10e54402b64" containerName="kserve-container"
Apr 16 17:05:27.546100 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:27.546030    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="fd9ce055-b7c9-4361-8416-f10e54402b64" containerName="kserve-container"
Apr 16 17:05:27.549175 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:27.549159    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft"
Apr 16 17:05:27.555956 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:27.555927    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft"]
Apr 16 17:05:27.661742 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:27.661719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft\" (UID: \"10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft"
Apr 16 17:05:27.762097 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:27.762043    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft\" (UID: \"10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft"
Apr 16 17:05:27.762338 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:27.762322    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft\" (UID: \"10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft"
Apr 16 17:05:27.859815 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:27.859790    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft"
Apr 16 17:05:27.975838 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:27.975808    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft"]
Apr 16 17:05:27.979327 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:05:27.979301    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10119a58_c4d6_4ce4_ab8e_d1ee2d934ad8.slice/crio-03bcdaf287ae88c9023c16cfde148d40e99cee0439a98d0b5191f16f98e89cf7 WatchSource:0}: Error finding container 03bcdaf287ae88c9023c16cfde148d40e99cee0439a98d0b5191f16f98e89cf7: Status 404 returned error can't find the container with id 03bcdaf287ae88c9023c16cfde148d40e99cee0439a98d0b5191f16f98e89cf7
Apr 16 17:05:28.770649 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:28.770606    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft" event={"ID":"10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8","Type":"ContainerStarted","Data":"6263fedd15d39df1ee6d3cdb4290ff082760e524a532fd7fe81effedd4063850"}
Apr 16 17:05:28.770649 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:28.770651    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft" event={"ID":"10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8","Type":"ContainerStarted","Data":"03bcdaf287ae88c9023c16cfde148d40e99cee0439a98d0b5191f16f98e89cf7"}
Apr 16 17:05:32.782751 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:32.782719    2578 generic.go:358] "Generic (PLEG): container finished" podID="10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8" containerID="6263fedd15d39df1ee6d3cdb4290ff082760e524a532fd7fe81effedd4063850" exitCode=0
Apr 16 17:05:32.783138 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:32.782794    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft" event={"ID":"10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8","Type":"ContainerDied","Data":"6263fedd15d39df1ee6d3cdb4290ff082760e524a532fd7fe81effedd4063850"}
Apr 16 17:05:33.787071 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:33.787035    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft" event={"ID":"10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8","Type":"ContainerStarted","Data":"bb3e745e277c4f3d922b5ba244476d123990162ce2ba165d3176fb5d690e82d4"}
Apr 16 17:05:33.787475 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:33.787256    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft"
Apr 16 17:05:33.802845 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:33.802806    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft" podStartSLOduration=6.802794806 podStartE2EDuration="6.802794806s" podCreationTimestamp="2026-04-16 17:05:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:05:33.801704175 +0000 UTC m=+2143.198920563" watchObservedRunningTime="2026-04-16 17:05:33.802794806 +0000 UTC m=+2143.200011193"
Apr 16 17:05:34.561709 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:34.561674    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l" podUID="de7978af-cbd9-4d2a-9ef1-33bc535c37f2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.49:8080: connect: connection refused"
Apr 16 17:05:34.717486 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:34.717466    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l"
Apr 16 17:05:34.791311 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:34.791232    2578 generic.go:358] "Generic (PLEG): container finished" podID="de7978af-cbd9-4d2a-9ef1-33bc535c37f2" containerID="807de508d0804cef24dfcac5f7a6a6d4eb07f158112cb3472ea49d33e9d29e7d" exitCode=0
Apr 16 17:05:34.791311 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:34.791299    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l"
Apr 16 17:05:34.791752 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:34.791316    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l" event={"ID":"de7978af-cbd9-4d2a-9ef1-33bc535c37f2","Type":"ContainerDied","Data":"807de508d0804cef24dfcac5f7a6a6d4eb07f158112cb3472ea49d33e9d29e7d"}
Apr 16 17:05:34.791752 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:34.791355    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l" event={"ID":"de7978af-cbd9-4d2a-9ef1-33bc535c37f2","Type":"ContainerDied","Data":"714f059be6bedec199fa9857e57533f82f2d7c670487564489e3efdcdfedf403"}
Apr 16 17:05:34.791752 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:34.791367    2578 scope.go:117] "RemoveContainer" containerID="807de508d0804cef24dfcac5f7a6a6d4eb07f158112cb3472ea49d33e9d29e7d"
Apr 16 17:05:34.798748 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:34.798720    2578 scope.go:117] "RemoveContainer" containerID="c9d1dca1aa6cec31cf71e007cd9004ecfce06bdbfad72c5ec8c3ebab79dcd3b4"
Apr 16 17:05:34.805527 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:34.805512    2578 scope.go:117] "RemoveContainer" containerID="807de508d0804cef24dfcac5f7a6a6d4eb07f158112cb3472ea49d33e9d29e7d"
Apr 16 17:05:34.805834 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:05:34.805814    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"807de508d0804cef24dfcac5f7a6a6d4eb07f158112cb3472ea49d33e9d29e7d\": container with ID starting with 807de508d0804cef24dfcac5f7a6a6d4eb07f158112cb3472ea49d33e9d29e7d not found: ID does not exist" containerID="807de508d0804cef24dfcac5f7a6a6d4eb07f158112cb3472ea49d33e9d29e7d"
Apr 16 17:05:34.805905 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:34.805839    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"807de508d0804cef24dfcac5f7a6a6d4eb07f158112cb3472ea49d33e9d29e7d"} err="failed to get container status \"807de508d0804cef24dfcac5f7a6a6d4eb07f158112cb3472ea49d33e9d29e7d\": rpc error: code = NotFound desc = could not find container \"807de508d0804cef24dfcac5f7a6a6d4eb07f158112cb3472ea49d33e9d29e7d\": container with ID starting with 807de508d0804cef24dfcac5f7a6a6d4eb07f158112cb3472ea49d33e9d29e7d not found: ID does not exist"
Apr 16 17:05:34.805905 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:34.805854    2578 scope.go:117] "RemoveContainer" containerID="c9d1dca1aa6cec31cf71e007cd9004ecfce06bdbfad72c5ec8c3ebab79dcd3b4"
Apr 16 17:05:34.806095 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:05:34.806078    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d1dca1aa6cec31cf71e007cd9004ecfce06bdbfad72c5ec8c3ebab79dcd3b4\": container with ID starting with c9d1dca1aa6cec31cf71e007cd9004ecfce06bdbfad72c5ec8c3ebab79dcd3b4 not found: ID does not exist" containerID="c9d1dca1aa6cec31cf71e007cd9004ecfce06bdbfad72c5ec8c3ebab79dcd3b4"
Apr 16 17:05:34.806139 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:34.806103    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d1dca1aa6cec31cf71e007cd9004ecfce06bdbfad72c5ec8c3ebab79dcd3b4"} err="failed to get container status \"c9d1dca1aa6cec31cf71e007cd9004ecfce06bdbfad72c5ec8c3ebab79dcd3b4\": rpc error: code = NotFound desc = could not find container \"c9d1dca1aa6cec31cf71e007cd9004ecfce06bdbfad72c5ec8c3ebab79dcd3b4\": container with ID starting with c9d1dca1aa6cec31cf71e007cd9004ecfce06bdbfad72c5ec8c3ebab79dcd3b4 not found: ID does not exist"
Apr 16 17:05:34.809381 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:34.809365    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de7978af-cbd9-4d2a-9ef1-33bc535c37f2-kserve-provision-location\") pod \"de7978af-cbd9-4d2a-9ef1-33bc535c37f2\" (UID: \"de7978af-cbd9-4d2a-9ef1-33bc535c37f2\") "
Apr 16 17:05:34.809649 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:34.809632    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7978af-cbd9-4d2a-9ef1-33bc535c37f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de7978af-cbd9-4d2a-9ef1-33bc535c37f2" (UID: "de7978af-cbd9-4d2a-9ef1-33bc535c37f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:05:34.910693 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:34.910673    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de7978af-cbd9-4d2a-9ef1-33bc535c37f2-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:05:35.115690 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:35.115657    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l"]
Apr 16 17:05:35.115808 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:35.115702    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-rth7l"]
Apr 16 17:05:35.234944 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:05:35.234914    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de7978af-cbd9-4d2a-9ef1-33bc535c37f2" path="/var/lib/kubelet/pods/de7978af-cbd9-4d2a-9ef1-33bc535c37f2/volumes"
Apr 16 17:06:04.793954 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:04.793914    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft" podUID="10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.50:8080: connect: connection refused"
Apr 16 17:06:14.792124 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:14.792087    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft" podUID="10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.50:8080: connect: connection refused"
Apr 16 17:06:24.792120 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:24.792076    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft" podUID="10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.50:8080: connect: connection refused"
Apr 16 17:06:34.792416 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:34.792378    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft" podUID="10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.50:8080: connect: connection refused"
Apr 16 17:06:44.796290 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:44.796256    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft"
Apr 16 17:06:47.726760 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:47.726729    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft"]
Apr 16 17:06:47.727139 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:47.726987    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft" podUID="10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8" containerName="kserve-container" containerID="cri-o://bb3e745e277c4f3d922b5ba244476d123990162ce2ba165d3176fb5d690e82d4" gracePeriod=30
Apr 16 17:06:49.822936 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:49.822905    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt"]
Apr 16 17:06:49.823286 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:49.823218    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de7978af-cbd9-4d2a-9ef1-33bc535c37f2" containerName="kserve-container"
Apr 16 17:06:49.823286 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:49.823228    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="de7978af-cbd9-4d2a-9ef1-33bc535c37f2" containerName="kserve-container"
Apr 16 17:06:49.823286 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:49.823245    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de7978af-cbd9-4d2a-9ef1-33bc535c37f2" containerName="storage-initializer"
Apr 16 17:06:49.823286 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:49.823251    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="de7978af-cbd9-4d2a-9ef1-33bc535c37f2" containerName="storage-initializer"
Apr 16 17:06:49.823422 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:49.823304    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="de7978af-cbd9-4d2a-9ef1-33bc535c37f2" containerName="kserve-container"
Apr 16 17:06:49.826276 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:49.826259    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt"
Apr 16 17:06:49.837633 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:49.837612    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt"]
Apr 16 17:06:49.945243 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:49.945219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd91422b-b698-43be-9407-4633ba0035f7-kserve-provision-location\") pod \"isvc-sklearn-predictor-7bbf8748f4-xjfqt\" (UID: \"cd91422b-b698-43be-9407-4633ba0035f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt"
Apr 16 17:06:50.045980 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:50.045957    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd91422b-b698-43be-9407-4633ba0035f7-kserve-provision-location\") pod \"isvc-sklearn-predictor-7bbf8748f4-xjfqt\" (UID: \"cd91422b-b698-43be-9407-4633ba0035f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt"
Apr 16 17:06:50.046272 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:50.046257    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd91422b-b698-43be-9407-4633ba0035f7-kserve-provision-location\") pod \"isvc-sklearn-predictor-7bbf8748f4-xjfqt\" (UID: \"cd91422b-b698-43be-9407-4633ba0035f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt"
Apr 16 17:06:50.136219 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:50.136162    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt"
Apr 16 17:06:50.251203 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:50.251167    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt"]
Apr 16 17:06:50.253690 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:06:50.253665    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd91422b_b698_43be_9407_4633ba0035f7.slice/crio-6dd83be142410d454675aaf43276f42f4b58c356f931ae322e3ff951e7c977e7 WatchSource:0}: Error finding container 6dd83be142410d454675aaf43276f42f4b58c356f931ae322e3ff951e7c977e7: Status 404 returned error can't find the container with id 6dd83be142410d454675aaf43276f42f4b58c356f931ae322e3ff951e7c977e7
Apr 16 17:06:51.008290 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:51.008257    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt" event={"ID":"cd91422b-b698-43be-9407-4633ba0035f7","Type":"ContainerStarted","Data":"0444fa8d0453addbfd5adcded00491c448efab1c22a0c65afaf5604613287e66"}
Apr 16 17:06:51.008290 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:51.008295    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt" event={"ID":"cd91422b-b698-43be-9407-4633ba0035f7","Type":"ContainerStarted","Data":"6dd83be142410d454675aaf43276f42f4b58c356f931ae322e3ff951e7c977e7"}
Apr 16 17:06:51.954725 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:51.954706    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft"
Apr 16 17:06:52.011801 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:52.011719    2578 generic.go:358] "Generic (PLEG): container finished" podID="10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8" containerID="bb3e745e277c4f3d922b5ba244476d123990162ce2ba165d3176fb5d690e82d4" exitCode=0
Apr 16 17:06:52.011801 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:52.011799    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft"
Apr 16 17:06:52.012240 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:52.011801    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft" event={"ID":"10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8","Type":"ContainerDied","Data":"bb3e745e277c4f3d922b5ba244476d123990162ce2ba165d3176fb5d690e82d4"}
Apr 16 17:06:52.012240 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:52.011838    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft" event={"ID":"10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8","Type":"ContainerDied","Data":"03bcdaf287ae88c9023c16cfde148d40e99cee0439a98d0b5191f16f98e89cf7"}
Apr 16 17:06:52.012240 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:52.011855    2578 scope.go:117] "RemoveContainer" containerID="bb3e745e277c4f3d922b5ba244476d123990162ce2ba165d3176fb5d690e82d4"
Apr 16 17:06:52.018973 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:52.018955    2578 scope.go:117] "RemoveContainer" containerID="6263fedd15d39df1ee6d3cdb4290ff082760e524a532fd7fe81effedd4063850"
Apr 16 17:06:52.025729 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:52.025714    2578 scope.go:117] "RemoveContainer" containerID="bb3e745e277c4f3d922b5ba244476d123990162ce2ba165d3176fb5d690e82d4"
Apr 16 17:06:52.025960 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:06:52.025940    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3e745e277c4f3d922b5ba244476d123990162ce2ba165d3176fb5d690e82d4\": container with ID starting with bb3e745e277c4f3d922b5ba244476d123990162ce2ba165d3176fb5d690e82d4 not found: ID does not exist" containerID="bb3e745e277c4f3d922b5ba244476d123990162ce2ba165d3176fb5d690e82d4"
Apr 16 17:06:52.030859 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:52.026326    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3e745e277c4f3d922b5ba244476d123990162ce2ba165d3176fb5d690e82d4"} err="failed to get container status \"bb3e745e277c4f3d922b5ba244476d123990162ce2ba165d3176fb5d690e82d4\": rpc error: code = NotFound desc = could not find container \"bb3e745e277c4f3d922b5ba244476d123990162ce2ba165d3176fb5d690e82d4\": container with ID starting with bb3e745e277c4f3d922b5ba244476d123990162ce2ba165d3176fb5d690e82d4 not found: ID does not exist"
Apr 16 17:06:52.030859 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:52.026404    2578 scope.go:117] "RemoveContainer" containerID="6263fedd15d39df1ee6d3cdb4290ff082760e524a532fd7fe81effedd4063850"
Apr 16 17:06:52.031225 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:06:52.031202    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6263fedd15d39df1ee6d3cdb4290ff082760e524a532fd7fe81effedd4063850\": container with ID starting with 6263fedd15d39df1ee6d3cdb4290ff082760e524a532fd7fe81effedd4063850 not found: ID does not exist" containerID="6263fedd15d39df1ee6d3cdb4290ff082760e524a532fd7fe81effedd4063850"
Apr 16 17:06:52.031292 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:52.031231    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6263fedd15d39df1ee6d3cdb4290ff082760e524a532fd7fe81effedd4063850"} err="failed to get container status \"6263fedd15d39df1ee6d3cdb4290ff082760e524a532fd7fe81effedd4063850\": rpc error: code = NotFound desc = could not find container \"6263fedd15d39df1ee6d3cdb4290ff082760e524a532fd7fe81effedd4063850\": container with ID starting with 6263fedd15d39df1ee6d3cdb4290ff082760e524a532fd7fe81effedd4063850 not found: ID does not exist"
Apr 16 17:06:52.059338 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:52.059314    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8-kserve-provision-location\") pod \"10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8\" (UID: \"10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8\") "
Apr 16 17:06:52.059662 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:52.059642    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8" (UID: "10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:06:52.160903 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:52.160875    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:06:52.331980 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:52.331955    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft"]
Apr 16 17:06:52.336206 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:52.336183    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-w59ft"]
Apr 16 17:06:53.235226 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:53.235193    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8" path="/var/lib/kubelet/pods/10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8/volumes"
Apr 16 17:06:54.019773 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:54.019746    2578 generic.go:358] "Generic (PLEG): container finished" podID="cd91422b-b698-43be-9407-4633ba0035f7" containerID="0444fa8d0453addbfd5adcded00491c448efab1c22a0c65afaf5604613287e66" exitCode=0
Apr 16 17:06:54.019943 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:54.019798    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt" event={"ID":"cd91422b-b698-43be-9407-4633ba0035f7","Type":"ContainerDied","Data":"0444fa8d0453addbfd5adcded00491c448efab1c22a0c65afaf5604613287e66"}
Apr 16 17:06:55.023848 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:55.023809    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt" event={"ID":"cd91422b-b698-43be-9407-4633ba0035f7","Type":"ContainerStarted","Data":"983219b54b0c036679a135744ad5256064103c4fa709890a5ebb1537bccb0a3e"}
Apr 16 17:06:55.024251 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:55.024157    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt"
Apr 16 17:06:55.025095 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:55.025068    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt" podUID="cd91422b-b698-43be-9407-4633ba0035f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Apr 16 17:06:55.065323 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:55.065285    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt" podStartSLOduration=6.065273695 podStartE2EDuration="6.065273695s" podCreationTimestamp="2026-04-16 17:06:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:06:55.062708423 +0000 UTC m=+2224.459924824" watchObservedRunningTime="2026-04-16 17:06:55.065273695 +0000 UTC m=+2224.462490082"
Apr 16 17:06:56.027326 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:06:56.027286    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt" podUID="cd91422b-b698-43be-9407-4633ba0035f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Apr 16 17:07:06.027499 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:07:06.027459    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt" podUID="cd91422b-b698-43be-9407-4633ba0035f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Apr 16 17:07:16.027329 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:07:16.027278    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt" podUID="cd91422b-b698-43be-9407-4633ba0035f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Apr 16 17:07:26.027620 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:07:26.027574    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt" podUID="cd91422b-b698-43be-9407-4633ba0035f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Apr 16 17:07:36.027458 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:07:36.027419    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt" podUID="cd91422b-b698-43be-9407-4633ba0035f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Apr 16 17:07:46.027630 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:07:46.027577    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt" podUID="cd91422b-b698-43be-9407-4633ba0035f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Apr 16 17:07:56.028416 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:07:56.028378    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt"
Apr 16 17:07:59.934130 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:07:59.934095    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt"]
Apr 16 17:07:59.934513 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:07:59.934363    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt" podUID="cd91422b-b698-43be-9407-4633ba0035f7" containerName="kserve-container" containerID="cri-o://983219b54b0c036679a135744ad5256064103c4fa709890a5ebb1537bccb0a3e" gracePeriod=30
Apr 16 17:08:00.022150 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:00.022109    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd"]
Apr 16 17:08:00.022514 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:00.022496    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8" containerName="kserve-container"
Apr 16 17:08:00.022685 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:00.022516    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8" containerName="kserve-container"
Apr 16 17:08:00.022685 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:00.022538    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8" containerName="storage-initializer"
Apr 16 17:08:00.022685 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:00.022547    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8" containerName="storage-initializer"
Apr 16 17:08:00.022685 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:00.022668    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="10119a58-c4d6-4ce4-ab8e-d1ee2d934ad8" containerName="kserve-container"
Apr 16 17:08:00.025816 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:00.025794    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd"
Apr 16 17:08:00.034337 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:00.034311    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd"]
Apr 16 17:08:00.136980 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:00.136940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d260934-784c-41d5-937b-f9877a05a8e3-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-695d5f5568-9zgdd\" (UID: \"3d260934-784c-41d5-937b-f9877a05a8e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd"
Apr 16 17:08:00.238296 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:00.238267    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d260934-784c-41d5-937b-f9877a05a8e3-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-695d5f5568-9zgdd\" (UID: \"3d260934-784c-41d5-937b-f9877a05a8e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd"
Apr 16 17:08:00.238646 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:00.238626    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d260934-784c-41d5-937b-f9877a05a8e3-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-695d5f5568-9zgdd\" (UID: \"3d260934-784c-41d5-937b-f9877a05a8e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd"
Apr 16 17:08:00.335864 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:00.335836    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd"
Apr 16 17:08:00.457678 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:00.457653    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd"]
Apr 16 17:08:00.460177 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:08:00.460148    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d260934_784c_41d5_937b_f9877a05a8e3.slice/crio-112c1c44da2b88d378db9eed323bc8811a7a827ff06e08c8da2ce2d6c4c1a43d WatchSource:0}: Error finding container 112c1c44da2b88d378db9eed323bc8811a7a827ff06e08c8da2ce2d6c4c1a43d: Status 404 returned error can't find the container with id 112c1c44da2b88d378db9eed323bc8811a7a827ff06e08c8da2ce2d6c4c1a43d
Apr 16 17:08:01.235310 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:01.235281    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd" event={"ID":"3d260934-784c-41d5-937b-f9877a05a8e3","Type":"ContainerStarted","Data":"30605a1d22220f587989184d71097e7f68c6dcb241b6c743058892676c99d7a4"}
Apr 16 17:08:01.235310 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:01.235314    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd" event={"ID":"3d260934-784c-41d5-937b-f9877a05a8e3","Type":"ContainerStarted","Data":"112c1c44da2b88d378db9eed323bc8811a7a827ff06e08c8da2ce2d6c4c1a43d"}
Apr 16 17:08:03.765497 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:03.765470    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt"
Apr 16 17:08:03.865352 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:03.865329    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd91422b-b698-43be-9407-4633ba0035f7-kserve-provision-location\") pod \"cd91422b-b698-43be-9407-4633ba0035f7\" (UID: \"cd91422b-b698-43be-9407-4633ba0035f7\") "
Apr 16 17:08:03.865649 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:03.865626    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd91422b-b698-43be-9407-4633ba0035f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd91422b-b698-43be-9407-4633ba0035f7" (UID: "cd91422b-b698-43be-9407-4633ba0035f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:08:03.966142 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:03.966119    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd91422b-b698-43be-9407-4633ba0035f7-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:08:04.239729 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:04.239667    2578 generic.go:358] "Generic (PLEG): container finished" podID="cd91422b-b698-43be-9407-4633ba0035f7" containerID="983219b54b0c036679a135744ad5256064103c4fa709890a5ebb1537bccb0a3e" exitCode=0
Apr 16 17:08:04.239839 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:04.239740    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt"
Apr 16 17:08:04.239839 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:04.239754    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt" event={"ID":"cd91422b-b698-43be-9407-4633ba0035f7","Type":"ContainerDied","Data":"983219b54b0c036679a135744ad5256064103c4fa709890a5ebb1537bccb0a3e"}
Apr 16 17:08:04.239839 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:04.239787    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt" event={"ID":"cd91422b-b698-43be-9407-4633ba0035f7","Type":"ContainerDied","Data":"6dd83be142410d454675aaf43276f42f4b58c356f931ae322e3ff951e7c977e7"}
Apr 16 17:08:04.239839 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:04.239802    2578 scope.go:117] "RemoveContainer" containerID="983219b54b0c036679a135744ad5256064103c4fa709890a5ebb1537bccb0a3e"
Apr 16 17:08:04.241423 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:04.241395    2578 generic.go:358] "Generic (PLEG): container finished" podID="3d260934-784c-41d5-937b-f9877a05a8e3" containerID="30605a1d22220f587989184d71097e7f68c6dcb241b6c743058892676c99d7a4" exitCode=0
Apr 16 17:08:04.241545 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:04.241436    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd" event={"ID":"3d260934-784c-41d5-937b-f9877a05a8e3","Type":"ContainerDied","Data":"30605a1d22220f587989184d71097e7f68c6dcb241b6c743058892676c99d7a4"}
Apr 16 17:08:04.247289 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:04.247269    2578 scope.go:117] "RemoveContainer" containerID="0444fa8d0453addbfd5adcded00491c448efab1c22a0c65afaf5604613287e66"
Apr 16 17:08:04.254048 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:04.254036    2578 scope.go:117] "RemoveContainer" containerID="983219b54b0c036679a135744ad5256064103c4fa709890a5ebb1537bccb0a3e"
Apr 16 17:08:04.254312 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:08:04.254293    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983219b54b0c036679a135744ad5256064103c4fa709890a5ebb1537bccb0a3e\": container with ID starting with 983219b54b0c036679a135744ad5256064103c4fa709890a5ebb1537bccb0a3e not found: ID does not exist" containerID="983219b54b0c036679a135744ad5256064103c4fa709890a5ebb1537bccb0a3e"
Apr 16 17:08:04.254363 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:04.254319    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983219b54b0c036679a135744ad5256064103c4fa709890a5ebb1537bccb0a3e"} err="failed to get container status \"983219b54b0c036679a135744ad5256064103c4fa709890a5ebb1537bccb0a3e\": rpc error: code = NotFound desc = could not find container \"983219b54b0c036679a135744ad5256064103c4fa709890a5ebb1537bccb0a3e\": container with ID starting with 983219b54b0c036679a135744ad5256064103c4fa709890a5ebb1537bccb0a3e not found: ID does not exist"
Apr 16 17:08:04.254363 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:04.254335    2578 scope.go:117] "RemoveContainer" containerID="0444fa8d0453addbfd5adcded00491c448efab1c22a0c65afaf5604613287e66"
Apr 16 17:08:04.254616 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:08:04.254599    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0444fa8d0453addbfd5adcded00491c448efab1c22a0c65afaf5604613287e66\": container with ID starting with 0444fa8d0453addbfd5adcded00491c448efab1c22a0c65afaf5604613287e66 not found: ID does not exist" containerID="0444fa8d0453addbfd5adcded00491c448efab1c22a0c65afaf5604613287e66"
Apr 16 17:08:04.254680 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:04.254619    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0444fa8d0453addbfd5adcded00491c448efab1c22a0c65afaf5604613287e66"} err="failed to get container status \"0444fa8d0453addbfd5adcded00491c448efab1c22a0c65afaf5604613287e66\": rpc error: code = NotFound desc = could not find container \"0444fa8d0453addbfd5adcded00491c448efab1c22a0c65afaf5604613287e66\": container with ID starting with 0444fa8d0453addbfd5adcded00491c448efab1c22a0c65afaf5604613287e66 not found: ID does not exist"
Apr 16 17:08:04.268700 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:04.268678    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt"]
Apr 16 17:08:04.272182 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:04.272160    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7bbf8748f4-xjfqt"]
Apr 16 17:08:05.235311 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:05.235279    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd91422b-b698-43be-9407-4633ba0035f7" path="/var/lib/kubelet/pods/cd91422b-b698-43be-9407-4633ba0035f7/volumes"
Apr 16 17:08:05.245100 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:05.245075    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd" event={"ID":"3d260934-784c-41d5-937b-f9877a05a8e3","Type":"ContainerStarted","Data":"924b10866310f21c1b1b825302454a4705b252c6dc448e668d3679a8606851b0"}
Apr 16 17:08:05.245285 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:05.245267    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd"
Apr 16 17:08:05.261616 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:05.261526    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd" podStartSLOduration=5.261511304 podStartE2EDuration="5.261511304s" podCreationTimestamp="2026-04-16 17:08:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:08:05.260829215 +0000 UTC m=+2294.658045602" watchObservedRunningTime="2026-04-16 17:08:05.261511304 +0000 UTC m=+2294.658727693"
Apr 16 17:08:36.330359 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:36.330318    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd" podUID="3d260934-784c-41d5-937b-f9877a05a8e3" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Apr 16 17:08:46.250689 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:46.250655    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd"
Apr 16 17:08:50.125252 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:50.125217    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd"]
Apr 16 17:08:50.125723 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:50.125477    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd" podUID="3d260934-784c-41d5-937b-f9877a05a8e3" containerName="kserve-container" containerID="cri-o://924b10866310f21c1b1b825302454a4705b252c6dc448e668d3679a8606851b0" gracePeriod=30
Apr 16 17:08:50.173384 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:50.173350    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv"]
Apr 16 17:08:50.173778 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:50.173760    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd91422b-b698-43be-9407-4633ba0035f7" containerName="kserve-container"
Apr 16 17:08:50.173778 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:50.173779    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd91422b-b698-43be-9407-4633ba0035f7" containerName="kserve-container"
Apr 16 17:08:50.173949 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:50.173792    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd91422b-b698-43be-9407-4633ba0035f7" containerName="storage-initializer"
Apr 16 17:08:50.173949 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:50.173800    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd91422b-b698-43be-9407-4633ba0035f7" containerName="storage-initializer"
Apr 16 17:08:50.173949 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:50.173873    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="cd91422b-b698-43be-9407-4633ba0035f7" containerName="kserve-container"
Apr 16 17:08:50.176848 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:50.176828    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv"
Apr 16 17:08:50.185917 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:50.185888    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv"]
Apr 16 17:08:50.284447 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:50.284425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/998b3b24-4b82-43af-b0a9-8f7d827cfedd-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-b7c8746d-gbtmv\" (UID: \"998b3b24-4b82-43af-b0a9-8f7d827cfedd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv"
Apr 16 17:08:50.385656 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:50.385585    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/998b3b24-4b82-43af-b0a9-8f7d827cfedd-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-b7c8746d-gbtmv\" (UID: \"998b3b24-4b82-43af-b0a9-8f7d827cfedd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv"
Apr 16 17:08:50.385966 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:50.385944    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/998b3b24-4b82-43af-b0a9-8f7d827cfedd-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-b7c8746d-gbtmv\" (UID: \"998b3b24-4b82-43af-b0a9-8f7d827cfedd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv"
Apr 16 17:08:50.486671 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:50.486644    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv"
Apr 16 17:08:50.599005 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:50.598976    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv"]
Apr 16 17:08:50.601885 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:08:50.601849    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998b3b24_4b82_43af_b0a9_8f7d827cfedd.slice/crio-2efb58ad5f818f284f5c8b4ac836df41d2eaf5ed1e10ed09cb86dd31d9dfb460 WatchSource:0}: Error finding container 2efb58ad5f818f284f5c8b4ac836df41d2eaf5ed1e10ed09cb86dd31d9dfb460: Status 404 returned error can't find the container with id 2efb58ad5f818f284f5c8b4ac836df41d2eaf5ed1e10ed09cb86dd31d9dfb460
Apr 16 17:08:51.370848 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:51.370815    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv" event={"ID":"998b3b24-4b82-43af-b0a9-8f7d827cfedd","Type":"ContainerStarted","Data":"201bf58b07463ed18b3108f490d05b325023f3ad92e43e95a8aba61bfd867d3a"}
Apr 16 17:08:51.370848 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:51.370849    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv" event={"ID":"998b3b24-4b82-43af-b0a9-8f7d827cfedd","Type":"ContainerStarted","Data":"2efb58ad5f818f284f5c8b4ac836df41d2eaf5ed1e10ed09cb86dd31d9dfb460"}
Apr 16 17:08:56.248750 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:56.248719    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd" podUID="3d260934-784c-41d5-937b-f9877a05a8e3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.52:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.134.0.52:8080: connect: connection refused"
Apr 16 17:08:56.386662 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:56.386629    2578 generic.go:358] "Generic (PLEG): container finished" podID="998b3b24-4b82-43af-b0a9-8f7d827cfedd" containerID="201bf58b07463ed18b3108f490d05b325023f3ad92e43e95a8aba61bfd867d3a" exitCode=0
Apr 16 17:08:56.386896 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:56.386704    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv" event={"ID":"998b3b24-4b82-43af-b0a9-8f7d827cfedd","Type":"ContainerDied","Data":"201bf58b07463ed18b3108f490d05b325023f3ad92e43e95a8aba61bfd867d3a"}
Apr 16 17:08:56.388424 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:56.388402    2578 generic.go:358] "Generic (PLEG): container finished" podID="3d260934-784c-41d5-937b-f9877a05a8e3" containerID="924b10866310f21c1b1b825302454a4705b252c6dc448e668d3679a8606851b0" exitCode=0
Apr 16 17:08:56.388523 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:56.388467    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd" event={"ID":"3d260934-784c-41d5-937b-f9877a05a8e3","Type":"ContainerDied","Data":"924b10866310f21c1b1b825302454a4705b252c6dc448e668d3679a8606851b0"}
Apr 16 17:08:56.463659 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:56.463633    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd"
Apr 16 17:08:56.541150 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:56.541070    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d260934-784c-41d5-937b-f9877a05a8e3-kserve-provision-location\") pod \"3d260934-784c-41d5-937b-f9877a05a8e3\" (UID: \"3d260934-784c-41d5-937b-f9877a05a8e3\") "
Apr 16 17:08:56.541366 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:56.541344    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d260934-784c-41d5-937b-f9877a05a8e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d260934-784c-41d5-937b-f9877a05a8e3" (UID: "3d260934-784c-41d5-937b-f9877a05a8e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:08:56.642459 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:56.642417    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d260934-784c-41d5-937b-f9877a05a8e3-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:08:57.393195 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:57.393162    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv" event={"ID":"998b3b24-4b82-43af-b0a9-8f7d827cfedd","Type":"ContainerStarted","Data":"1e1d97ba0001c9a59493e2709e08a1fd0e002debf19584ff4c9376aa8f46a69d"}
Apr 16 17:08:57.393656 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:57.393457    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv"
Apr 16 17:08:57.394707 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:57.394678    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv" podUID="998b3b24-4b82-43af-b0a9-8f7d827cfedd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Apr 16 17:08:57.394878 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:57.394854    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd" event={"ID":"3d260934-784c-41d5-937b-f9877a05a8e3","Type":"ContainerDied","Data":"112c1c44da2b88d378db9eed323bc8811a7a827ff06e08c8da2ce2d6c4c1a43d"}
Apr 16 17:08:57.394931 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:57.394874    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd"
Apr 16 17:08:57.394931 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:57.394901    2578 scope.go:117] "RemoveContainer" containerID="924b10866310f21c1b1b825302454a4705b252c6dc448e668d3679a8606851b0"
Apr 16 17:08:57.402789 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:57.402772    2578 scope.go:117] "RemoveContainer" containerID="30605a1d22220f587989184d71097e7f68c6dcb241b6c743058892676c99d7a4"
Apr 16 17:08:57.410734 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:57.410698    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv" podStartSLOduration=7.410686515 podStartE2EDuration="7.410686515s" podCreationTimestamp="2026-04-16 17:08:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:08:57.40985027 +0000 UTC m=+2346.807066658" watchObservedRunningTime="2026-04-16 17:08:57.410686515 +0000 UTC m=+2346.807902931"
Apr 16 17:08:57.422837 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:57.422815    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd"]
Apr 16 17:08:57.425003 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:57.424980    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-9zgdd"]
Apr 16 17:08:58.399181 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:58.399140    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv" podUID="998b3b24-4b82-43af-b0a9-8f7d827cfedd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Apr 16 17:08:59.236752 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:08:59.236722    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d260934-784c-41d5-937b-f9877a05a8e3" path="/var/lib/kubelet/pods/3d260934-784c-41d5-937b-f9877a05a8e3/volumes"
Apr 16 17:09:08.400007 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:08.399965    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv" podUID="998b3b24-4b82-43af-b0a9-8f7d827cfedd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Apr 16 17:09:18.400377 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:18.400347    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv"
Apr 16 17:09:27.215076 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.215050    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-b7c8746d-gbtmv_998b3b24-4b82-43af-b0a9-8f7d827cfedd/kserve-container/0.log"
Apr 16 17:09:27.336239 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.336204    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv"]
Apr 16 17:09:27.336467 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.336445    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv" podUID="998b3b24-4b82-43af-b0a9-8f7d827cfedd" containerName="kserve-container" containerID="cri-o://1e1d97ba0001c9a59493e2709e08a1fd0e002debf19584ff4c9376aa8f46a69d" gracePeriod=30
Apr 16 17:09:27.403214 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.403181    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt"]
Apr 16 17:09:27.403509 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.403497    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d260934-784c-41d5-937b-f9877a05a8e3" containerName="kserve-container"
Apr 16 17:09:27.403582 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.403511    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d260934-784c-41d5-937b-f9877a05a8e3" containerName="kserve-container"
Apr 16 17:09:27.403582 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.403530    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d260934-784c-41d5-937b-f9877a05a8e3" containerName="storage-initializer"
Apr 16 17:09:27.403582 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.403536    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d260934-784c-41d5-937b-f9877a05a8e3" containerName="storage-initializer"
Apr 16 17:09:27.403689 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.403621    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="3d260934-784c-41d5-937b-f9877a05a8e3" containerName="kserve-container"
Apr 16 17:09:27.405716 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.405701    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt"
Apr 16 17:09:27.413527 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.413502    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt"]
Apr 16 17:09:27.465196 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.465128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc5a814f-f66b-4ada-8455-3134740eb59a-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt\" (UID: \"fc5a814f-f66b-4ada-8455-3134740eb59a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt"
Apr 16 17:09:27.565507 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.565480    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc5a814f-f66b-4ada-8455-3134740eb59a-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt\" (UID: \"fc5a814f-f66b-4ada-8455-3134740eb59a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt"
Apr 16 17:09:27.565812 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.565793    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc5a814f-f66b-4ada-8455-3134740eb59a-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt\" (UID: \"fc5a814f-f66b-4ada-8455-3134740eb59a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt"
Apr 16 17:09:27.716331 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.716250    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt"
Apr 16 17:09:27.860321 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.860287    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt"]
Apr 16 17:09:27.864989 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:09:27.864936    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5a814f_f66b_4ada_8455_3134740eb59a.slice/crio-ff941dff3208c1a4fd2b76d37b35daa68ba7fab5092d0e40adbcd800f5117016 WatchSource:0}: Error finding container ff941dff3208c1a4fd2b76d37b35daa68ba7fab5092d0e40adbcd800f5117016: Status 404 returned error can't find the container with id ff941dff3208c1a4fd2b76d37b35daa68ba7fab5092d0e40adbcd800f5117016
Apr 16 17:09:27.866757 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:27.866733    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 17:09:28.261598 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.261542    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv"
Apr 16 17:09:28.373550 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.373466    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/998b3b24-4b82-43af-b0a9-8f7d827cfedd-kserve-provision-location\") pod \"998b3b24-4b82-43af-b0a9-8f7d827cfedd\" (UID: \"998b3b24-4b82-43af-b0a9-8f7d827cfedd\") "
Apr 16 17:09:28.388180 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.388151    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998b3b24-4b82-43af-b0a9-8f7d827cfedd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "998b3b24-4b82-43af-b0a9-8f7d827cfedd" (UID: "998b3b24-4b82-43af-b0a9-8f7d827cfedd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:09:28.474474 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.474441    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/998b3b24-4b82-43af-b0a9-8f7d827cfedd-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:09:28.484051 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.484023    2578 generic.go:358] "Generic (PLEG): container finished" podID="998b3b24-4b82-43af-b0a9-8f7d827cfedd" containerID="1e1d97ba0001c9a59493e2709e08a1fd0e002debf19584ff4c9376aa8f46a69d" exitCode=0
Apr 16 17:09:28.484185 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.484111    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv"
Apr 16 17:09:28.484241 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.484106    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv" event={"ID":"998b3b24-4b82-43af-b0a9-8f7d827cfedd","Type":"ContainerDied","Data":"1e1d97ba0001c9a59493e2709e08a1fd0e002debf19584ff4c9376aa8f46a69d"}
Apr 16 17:09:28.484241 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.484224    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv" event={"ID":"998b3b24-4b82-43af-b0a9-8f7d827cfedd","Type":"ContainerDied","Data":"2efb58ad5f818f284f5c8b4ac836df41d2eaf5ed1e10ed09cb86dd31d9dfb460"}
Apr 16 17:09:28.484333 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.484245    2578 scope.go:117] "RemoveContainer" containerID="1e1d97ba0001c9a59493e2709e08a1fd0e002debf19584ff4c9376aa8f46a69d"
Apr 16 17:09:28.485540 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.485517    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt" event={"ID":"fc5a814f-f66b-4ada-8455-3134740eb59a","Type":"ContainerStarted","Data":"c8abfc0ab16948463888548388356e0e537e1a2680a588be13e812a3995deb8b"}
Apr 16 17:09:28.485668 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.485549    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt" event={"ID":"fc5a814f-f66b-4ada-8455-3134740eb59a","Type":"ContainerStarted","Data":"ff941dff3208c1a4fd2b76d37b35daa68ba7fab5092d0e40adbcd800f5117016"}
Apr 16 17:09:28.492100 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.492079    2578 scope.go:117] "RemoveContainer" containerID="201bf58b07463ed18b3108f490d05b325023f3ad92e43e95a8aba61bfd867d3a"
Apr 16 17:09:28.498987 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.498972    2578 scope.go:117] "RemoveContainer" containerID="1e1d97ba0001c9a59493e2709e08a1fd0e002debf19584ff4c9376aa8f46a69d"
Apr 16 17:09:28.499228 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:09:28.499210    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e1d97ba0001c9a59493e2709e08a1fd0e002debf19584ff4c9376aa8f46a69d\": container with ID starting with 1e1d97ba0001c9a59493e2709e08a1fd0e002debf19584ff4c9376aa8f46a69d not found: ID does not exist" containerID="1e1d97ba0001c9a59493e2709e08a1fd0e002debf19584ff4c9376aa8f46a69d"
Apr 16 17:09:28.499299 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.499238    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e1d97ba0001c9a59493e2709e08a1fd0e002debf19584ff4c9376aa8f46a69d"} err="failed to get container status \"1e1d97ba0001c9a59493e2709e08a1fd0e002debf19584ff4c9376aa8f46a69d\": rpc error: code = NotFound desc = could not find container \"1e1d97ba0001c9a59493e2709e08a1fd0e002debf19584ff4c9376aa8f46a69d\": container with ID starting with 1e1d97ba0001c9a59493e2709e08a1fd0e002debf19584ff4c9376aa8f46a69d not found: ID does not exist"
Apr 16 17:09:28.499299 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.499262    2578 scope.go:117] "RemoveContainer" containerID="201bf58b07463ed18b3108f490d05b325023f3ad92e43e95a8aba61bfd867d3a"
Apr 16 17:09:28.499484 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:09:28.499467    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201bf58b07463ed18b3108f490d05b325023f3ad92e43e95a8aba61bfd867d3a\": container with ID starting with 201bf58b07463ed18b3108f490d05b325023f3ad92e43e95a8aba61bfd867d3a not found: ID does not exist" containerID="201bf58b07463ed18b3108f490d05b325023f3ad92e43e95a8aba61bfd867d3a"
Apr 16 17:09:28.499531 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.499489    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201bf58b07463ed18b3108f490d05b325023f3ad92e43e95a8aba61bfd867d3a"} err="failed to get container status \"201bf58b07463ed18b3108f490d05b325023f3ad92e43e95a8aba61bfd867d3a\": rpc error: code = NotFound desc = could not find container \"201bf58b07463ed18b3108f490d05b325023f3ad92e43e95a8aba61bfd867d3a\": container with ID starting with 201bf58b07463ed18b3108f490d05b325023f3ad92e43e95a8aba61bfd867d3a not found: ID does not exist"
Apr 16 17:09:28.516275 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.516253    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv"]
Apr 16 17:09:28.518447 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:28.518428    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b7c8746d-gbtmv"]
Apr 16 17:09:29.235565 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:29.235511    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="998b3b24-4b82-43af-b0a9-8f7d827cfedd" path="/var/lib/kubelet/pods/998b3b24-4b82-43af-b0a9-8f7d827cfedd/volumes"
Apr 16 17:09:32.499409 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:32.499380    2578 generic.go:358] "Generic (PLEG): container finished" podID="fc5a814f-f66b-4ada-8455-3134740eb59a" containerID="c8abfc0ab16948463888548388356e0e537e1a2680a588be13e812a3995deb8b" exitCode=0
Apr 16 17:09:32.499756 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:32.499417    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt" event={"ID":"fc5a814f-f66b-4ada-8455-3134740eb59a","Type":"ContainerDied","Data":"c8abfc0ab16948463888548388356e0e537e1a2680a588be13e812a3995deb8b"}
Apr 16 17:09:33.503688 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:33.503652    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt" event={"ID":"fc5a814f-f66b-4ada-8455-3134740eb59a","Type":"ContainerStarted","Data":"4f4712a84e772f7b638415d0f90ffb99c271db397ba98c6fefd923789d265512"}
Apr 16 17:09:33.504142 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:33.503871    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt"
Apr 16 17:09:33.523968 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:33.523914    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt" podStartSLOduration=6.523901442 podStartE2EDuration="6.523901442s" podCreationTimestamp="2026-04-16 17:09:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:09:33.521540082 +0000 UTC m=+2382.918756460" watchObservedRunningTime="2026-04-16 17:09:33.523901442 +0000 UTC m=+2382.921117895"
Apr 16 17:09:51.290676 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:51.290649    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 17:09:51.295986 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:09:51.295966    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 17:10:04.529850 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:04.529806    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt" podUID="fc5a814f-f66b-4ada-8455-3134740eb59a" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Apr 16 17:10:14.508804 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:14.508776    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt"
Apr 16 17:10:17.536340 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:17.536299    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt"]
Apr 16 17:10:17.536801 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:17.536592    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt" podUID="fc5a814f-f66b-4ada-8455-3134740eb59a" containerName="kserve-container" containerID="cri-o://4f4712a84e772f7b638415d0f90ffb99c271db397ba98c6fefd923789d265512" gracePeriod=30
Apr 16 17:10:17.594963 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:17.594930    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts"]
Apr 16 17:10:17.595366 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:17.595348    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="998b3b24-4b82-43af-b0a9-8f7d827cfedd" containerName="kserve-container"
Apr 16 17:10:17.595417 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:17.595370    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="998b3b24-4b82-43af-b0a9-8f7d827cfedd" containerName="kserve-container"
Apr 16 17:10:17.595417 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:17.595407    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="998b3b24-4b82-43af-b0a9-8f7d827cfedd" containerName="storage-initializer"
Apr 16 17:10:17.595485 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:17.595416    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="998b3b24-4b82-43af-b0a9-8f7d827cfedd" containerName="storage-initializer"
Apr 16 17:10:17.595518 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:17.595493    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="998b3b24-4b82-43af-b0a9-8f7d827cfedd" containerName="kserve-container"
Apr 16 17:10:17.598056 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:17.598040    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts"
Apr 16 17:10:17.606672 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:17.606651    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts"]
Apr 16 17:10:17.712320 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:17.712298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b001161-e5a6-4bc9-8285-50f6ee6227ef-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-9bcff48d6-4wvts\" (UID: \"2b001161-e5a6-4bc9-8285-50f6ee6227ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts"
Apr 16 17:10:17.812821 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:17.812766    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b001161-e5a6-4bc9-8285-50f6ee6227ef-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-9bcff48d6-4wvts\" (UID: \"2b001161-e5a6-4bc9-8285-50f6ee6227ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts"
Apr 16 17:10:17.813111 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:17.813094    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b001161-e5a6-4bc9-8285-50f6ee6227ef-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-9bcff48d6-4wvts\" (UID: \"2b001161-e5a6-4bc9-8285-50f6ee6227ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts"
Apr 16 17:10:17.908926 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:17.908898    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts"
Apr 16 17:10:18.050423 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:18.050363    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts"]
Apr 16 17:10:18.053232 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:10:18.053206    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b001161_e5a6_4bc9_8285_50f6ee6227ef.slice/crio-aa915c57ffba391e99d58a3b5d21a8b83a6e52dd64678bc218fb3399c45d4989 WatchSource:0}: Error finding container aa915c57ffba391e99d58a3b5d21a8b83a6e52dd64678bc218fb3399c45d4989: Status 404 returned error can't find the container with id aa915c57ffba391e99d58a3b5d21a8b83a6e52dd64678bc218fb3399c45d4989
Apr 16 17:10:18.632703 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:18.632669    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts" event={"ID":"2b001161-e5a6-4bc9-8285-50f6ee6227ef","Type":"ContainerStarted","Data":"a74c8b4db0b297a9d02b41fc30983a13ee246673190dde05ce3e613cb7128502"}
Apr 16 17:10:18.632703 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:18.632707    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts" event={"ID":"2b001161-e5a6-4bc9-8285-50f6ee6227ef","Type":"ContainerStarted","Data":"aa915c57ffba391e99d58a3b5d21a8b83a6e52dd64678bc218fb3399c45d4989"}
Apr 16 17:10:22.647880 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:22.647845    2578 generic.go:358] "Generic (PLEG): container finished" podID="2b001161-e5a6-4bc9-8285-50f6ee6227ef" containerID="a74c8b4db0b297a9d02b41fc30983a13ee246673190dde05ce3e613cb7128502" exitCode=0
Apr 16 17:10:22.648300 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:22.647920    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts" event={"ID":"2b001161-e5a6-4bc9-8285-50f6ee6227ef","Type":"ContainerDied","Data":"a74c8b4db0b297a9d02b41fc30983a13ee246673190dde05ce3e613cb7128502"}
Apr 16 17:10:23.653191 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:23.653157    2578 generic.go:358] "Generic (PLEG): container finished" podID="fc5a814f-f66b-4ada-8455-3134740eb59a" containerID="4f4712a84e772f7b638415d0f90ffb99c271db397ba98c6fefd923789d265512" exitCode=0
Apr 16 17:10:23.653585 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:23.653235    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt" event={"ID":"fc5a814f-f66b-4ada-8455-3134740eb59a","Type":"ContainerDied","Data":"4f4712a84e772f7b638415d0f90ffb99c271db397ba98c6fefd923789d265512"}
Apr 16 17:10:23.654931 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:23.654913    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts" event={"ID":"2b001161-e5a6-4bc9-8285-50f6ee6227ef","Type":"ContainerStarted","Data":"bf9a8abe19c21b55d432c318b5e6a2378b1de630e0df7783aa88f1dc30b780eb"}
Apr 16 17:10:23.655174 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:23.655152    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts"
Apr 16 17:10:23.656319 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:23.656299    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts" podUID="2b001161-e5a6-4bc9-8285-50f6ee6227ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Apr 16 17:10:23.668567 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:23.668538    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt"
Apr 16 17:10:23.670429 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:23.670393    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts" podStartSLOduration=6.670382512 podStartE2EDuration="6.670382512s" podCreationTimestamp="2026-04-16 17:10:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:10:23.669376757 +0000 UTC m=+2433.066593171" watchObservedRunningTime="2026-04-16 17:10:23.670382512 +0000 UTC m=+2433.067598900"
Apr 16 17:10:23.761674 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:23.761651    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc5a814f-f66b-4ada-8455-3134740eb59a-kserve-provision-location\") pod \"fc5a814f-f66b-4ada-8455-3134740eb59a\" (UID: \"fc5a814f-f66b-4ada-8455-3134740eb59a\") "
Apr 16 17:10:23.761960 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:23.761937    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5a814f-f66b-4ada-8455-3134740eb59a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc5a814f-f66b-4ada-8455-3134740eb59a" (UID: "fc5a814f-f66b-4ada-8455-3134740eb59a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:10:23.863017 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:23.862962    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc5a814f-f66b-4ada-8455-3134740eb59a-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:10:24.659075 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:24.659050    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt"
Apr 16 17:10:24.659514 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:24.659079    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt" event={"ID":"fc5a814f-f66b-4ada-8455-3134740eb59a","Type":"ContainerDied","Data":"ff941dff3208c1a4fd2b76d37b35daa68ba7fab5092d0e40adbcd800f5117016"}
Apr 16 17:10:24.659514 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:24.659120    2578 scope.go:117] "RemoveContainer" containerID="4f4712a84e772f7b638415d0f90ffb99c271db397ba98c6fefd923789d265512"
Apr 16 17:10:24.659514 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:24.659456    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts" podUID="2b001161-e5a6-4bc9-8285-50f6ee6227ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Apr 16 17:10:24.667436 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:24.667407    2578 scope.go:117] "RemoveContainer" containerID="c8abfc0ab16948463888548388356e0e537e1a2680a588be13e812a3995deb8b"
Apr 16 17:10:24.678258 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:24.678239    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt"]
Apr 16 17:10:24.681437 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:24.681417    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8wvxt"]
Apr 16 17:10:25.235517 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:25.235480    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc5a814f-f66b-4ada-8455-3134740eb59a" path="/var/lib/kubelet/pods/fc5a814f-f66b-4ada-8455-3134740eb59a/volumes"
Apr 16 17:10:34.659592 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:34.659537    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts" podUID="2b001161-e5a6-4bc9-8285-50f6ee6227ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Apr 16 17:10:44.659818 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:44.659773    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts" podUID="2b001161-e5a6-4bc9-8285-50f6ee6227ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Apr 16 17:10:54.659940 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:10:54.659899    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts" podUID="2b001161-e5a6-4bc9-8285-50f6ee6227ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Apr 16 17:11:04.659689 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:04.659649    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts" podUID="2b001161-e5a6-4bc9-8285-50f6ee6227ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Apr 16 17:11:14.659628 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:14.659587    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts" podUID="2b001161-e5a6-4bc9-8285-50f6ee6227ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Apr 16 17:11:24.661140 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:24.661111    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts"
Apr 16 17:11:27.775801 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:27.771958    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts"]
Apr 16 17:11:27.775801 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:27.772409    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts" podUID="2b001161-e5a6-4bc9-8285-50f6ee6227ef" containerName="kserve-container" containerID="cri-o://bf9a8abe19c21b55d432c318b5e6a2378b1de630e0df7783aa88f1dc30b780eb" gracePeriod=30
Apr 16 17:11:27.894441 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:27.894410    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx"]
Apr 16 17:11:27.894946 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:27.894926    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc5a814f-f66b-4ada-8455-3134740eb59a" containerName="storage-initializer"
Apr 16 17:11:27.895006 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:27.894950    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc5a814f-f66b-4ada-8455-3134740eb59a" containerName="storage-initializer"
Apr 16 17:11:27.895006 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:27.894975    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc5a814f-f66b-4ada-8455-3134740eb59a" containerName="kserve-container"
Apr 16 17:11:27.895006 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:27.894984    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc5a814f-f66b-4ada-8455-3134740eb59a" containerName="kserve-container"
Apr 16 17:11:27.895098 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:27.895064    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="fc5a814f-f66b-4ada-8455-3134740eb59a" containerName="kserve-container"
Apr 16 17:11:27.897125 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:27.897109    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx"
Apr 16 17:11:27.911417 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:27.911383    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx"]
Apr 16 17:11:27.999854 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:27.999828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/847899df-39aa-4342-917e-ba98d9201dc0-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx\" (UID: \"847899df-39aa-4342-917e-ba98d9201dc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx"
Apr 16 17:11:28.100804 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:28.100746    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/847899df-39aa-4342-917e-ba98d9201dc0-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx\" (UID: \"847899df-39aa-4342-917e-ba98d9201dc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx"
Apr 16 17:11:28.101077 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:28.101060    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/847899df-39aa-4342-917e-ba98d9201dc0-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx\" (UID: \"847899df-39aa-4342-917e-ba98d9201dc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx"
Apr 16 17:11:28.206780 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:28.206758    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx"
Apr 16 17:11:28.321620 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:28.321590    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx"]
Apr 16 17:11:28.835864 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:28.835832    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx" event={"ID":"847899df-39aa-4342-917e-ba98d9201dc0","Type":"ContainerStarted","Data":"bc0b9b7590bb1c8f638760894e23bdc0d7e217cdbbb18801fba3d84800421f50"}
Apr 16 17:11:28.835864 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:28.835867    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx" event={"ID":"847899df-39aa-4342-917e-ba98d9201dc0","Type":"ContainerStarted","Data":"7a6f1ca9c52bf485c7403cd3e7c24fb8fc77c356755c88d1c51b4511af9d2759"}
Apr 16 17:11:31.507073 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:31.507052    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts"
Apr 16 17:11:31.631797 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:31.631734    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b001161-e5a6-4bc9-8285-50f6ee6227ef-kserve-provision-location\") pod \"2b001161-e5a6-4bc9-8285-50f6ee6227ef\" (UID: \"2b001161-e5a6-4bc9-8285-50f6ee6227ef\") "
Apr 16 17:11:31.632020 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:31.631999    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b001161-e5a6-4bc9-8285-50f6ee6227ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b001161-e5a6-4bc9-8285-50f6ee6227ef" (UID: "2b001161-e5a6-4bc9-8285-50f6ee6227ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:11:31.732704 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:31.732679    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b001161-e5a6-4bc9-8285-50f6ee6227ef-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:11:31.844415 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:31.844386    2578 generic.go:358] "Generic (PLEG): container finished" podID="2b001161-e5a6-4bc9-8285-50f6ee6227ef" containerID="bf9a8abe19c21b55d432c318b5e6a2378b1de630e0df7783aa88f1dc30b780eb" exitCode=0
Apr 16 17:11:31.844531 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:31.844498    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts"
Apr 16 17:11:31.844531 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:31.844503    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts" event={"ID":"2b001161-e5a6-4bc9-8285-50f6ee6227ef","Type":"ContainerDied","Data":"bf9a8abe19c21b55d432c318b5e6a2378b1de630e0df7783aa88f1dc30b780eb"}
Apr 16 17:11:31.844635 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:31.844543    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts" event={"ID":"2b001161-e5a6-4bc9-8285-50f6ee6227ef","Type":"ContainerDied","Data":"aa915c57ffba391e99d58a3b5d21a8b83a6e52dd64678bc218fb3399c45d4989"}
Apr 16 17:11:31.844635 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:31.844583    2578 scope.go:117] "RemoveContainer" containerID="bf9a8abe19c21b55d432c318b5e6a2378b1de630e0df7783aa88f1dc30b780eb"
Apr 16 17:11:31.853138 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:31.853121    2578 scope.go:117] "RemoveContainer" containerID="a74c8b4db0b297a9d02b41fc30983a13ee246673190dde05ce3e613cb7128502"
Apr 16 17:11:31.859601 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:31.859586    2578 scope.go:117] "RemoveContainer" containerID="bf9a8abe19c21b55d432c318b5e6a2378b1de630e0df7783aa88f1dc30b780eb"
Apr 16 17:11:31.859827 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:11:31.859811    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf9a8abe19c21b55d432c318b5e6a2378b1de630e0df7783aa88f1dc30b780eb\": container with ID starting with bf9a8abe19c21b55d432c318b5e6a2378b1de630e0df7783aa88f1dc30b780eb not found: ID does not exist" containerID="bf9a8abe19c21b55d432c318b5e6a2378b1de630e0df7783aa88f1dc30b780eb"
Apr 16 17:11:31.859879 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:31.859834    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf9a8abe19c21b55d432c318b5e6a2378b1de630e0df7783aa88f1dc30b780eb"} err="failed to get container status \"bf9a8abe19c21b55d432c318b5e6a2378b1de630e0df7783aa88f1dc30b780eb\": rpc error: code = NotFound desc = could not find container \"bf9a8abe19c21b55d432c318b5e6a2378b1de630e0df7783aa88f1dc30b780eb\": container with ID starting with bf9a8abe19c21b55d432c318b5e6a2378b1de630e0df7783aa88f1dc30b780eb not found: ID does not exist"
Apr 16 17:11:31.859879 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:31.859849    2578 scope.go:117] "RemoveContainer" containerID="a74c8b4db0b297a9d02b41fc30983a13ee246673190dde05ce3e613cb7128502"
Apr 16 17:11:31.860080 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:11:31.860060    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a74c8b4db0b297a9d02b41fc30983a13ee246673190dde05ce3e613cb7128502\": container with ID starting with a74c8b4db0b297a9d02b41fc30983a13ee246673190dde05ce3e613cb7128502 not found: ID does not exist" containerID="a74c8b4db0b297a9d02b41fc30983a13ee246673190dde05ce3e613cb7128502"
Apr 16 17:11:31.860138 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:31.860090    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74c8b4db0b297a9d02b41fc30983a13ee246673190dde05ce3e613cb7128502"} err="failed to get container status \"a74c8b4db0b297a9d02b41fc30983a13ee246673190dde05ce3e613cb7128502\": rpc error: code = NotFound desc = could not find container \"a74c8b4db0b297a9d02b41fc30983a13ee246673190dde05ce3e613cb7128502\": container with ID starting with a74c8b4db0b297a9d02b41fc30983a13ee246673190dde05ce3e613cb7128502 not found: ID does not exist"
Apr 16 17:11:31.864492 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:31.864470    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts"]
Apr 16 17:11:31.870672 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:31.870639    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9bcff48d6-4wvts"]
Apr 16 17:11:32.848285 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:32.848254    2578 generic.go:358] "Generic (PLEG): container finished" podID="847899df-39aa-4342-917e-ba98d9201dc0" containerID="bc0b9b7590bb1c8f638760894e23bdc0d7e217cdbbb18801fba3d84800421f50" exitCode=0
Apr 16 17:11:32.848661 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:32.848328    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx" event={"ID":"847899df-39aa-4342-917e-ba98d9201dc0","Type":"ContainerDied","Data":"bc0b9b7590bb1c8f638760894e23bdc0d7e217cdbbb18801fba3d84800421f50"}
Apr 16 17:11:33.234978 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:33.234947    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b001161-e5a6-4bc9-8285-50f6ee6227ef" path="/var/lib/kubelet/pods/2b001161-e5a6-4bc9-8285-50f6ee6227ef/volumes"
Apr 16 17:11:33.854178 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:33.854146    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx" event={"ID":"847899df-39aa-4342-917e-ba98d9201dc0","Type":"ContainerStarted","Data":"99615add3ddb639e08f1a0ff0af21d9e54697cdb8f7713f1c8396e8c312affa5"}
Apr 16 17:11:33.854571 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:33.854430    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx"
Apr 16 17:11:33.855734 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:33.855709    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx" podUID="847899df-39aa-4342-917e-ba98d9201dc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Apr 16 17:11:33.870088 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:33.870046    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx" podStartSLOduration=6.8700341940000005 podStartE2EDuration="6.870034194s" podCreationTimestamp="2026-04-16 17:11:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:11:33.868294276 +0000 UTC m=+2503.265510664" watchObservedRunningTime="2026-04-16 17:11:33.870034194 +0000 UTC m=+2503.267250582"
Apr 16 17:11:34.857487 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:34.857452    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx" podUID="847899df-39aa-4342-917e-ba98d9201dc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Apr 16 17:11:44.858070 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:44.858033    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx" podUID="847899df-39aa-4342-917e-ba98d9201dc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Apr 16 17:11:54.857721 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:11:54.857680    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx" podUID="847899df-39aa-4342-917e-ba98d9201dc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Apr 16 17:12:04.857669 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:04.857624    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx" podUID="847899df-39aa-4342-917e-ba98d9201dc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Apr 16 17:12:14.857769 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:14.857729    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx" podUID="847899df-39aa-4342-917e-ba98d9201dc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Apr 16 17:12:24.857894 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:24.857850    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx" podUID="847899df-39aa-4342-917e-ba98d9201dc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Apr 16 17:12:34.858708 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:34.858678    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx"
Apr 16 17:12:37.978291 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:37.978250    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx"]
Apr 16 17:12:37.978665 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:37.978509    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx" podUID="847899df-39aa-4342-917e-ba98d9201dc0" containerName="kserve-container" containerID="cri-o://99615add3ddb639e08f1a0ff0af21d9e54697cdb8f7713f1c8396e8c312affa5" gracePeriod=30
Apr 16 17:12:38.023396 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:38.023344    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd"]
Apr 16 17:12:38.023679 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:38.023666    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b001161-e5a6-4bc9-8285-50f6ee6227ef" containerName="storage-initializer"
Apr 16 17:12:38.023729 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:38.023680    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b001161-e5a6-4bc9-8285-50f6ee6227ef" containerName="storage-initializer"
Apr 16 17:12:38.023729 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:38.023693    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b001161-e5a6-4bc9-8285-50f6ee6227ef" containerName="kserve-container"
Apr 16 17:12:38.023729 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:38.023699    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b001161-e5a6-4bc9-8285-50f6ee6227ef" containerName="kserve-container"
Apr 16 17:12:38.023828 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:38.023756    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="2b001161-e5a6-4bc9-8285-50f6ee6227ef" containerName="kserve-container"
Apr 16 17:12:38.026764 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:38.026749    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd"
Apr 16 17:12:38.034222 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:38.034199    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd"]
Apr 16 17:12:38.089045 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:38.089022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6305ba5b-257f-4df4-ba85-b9c2bb299358-kserve-provision-location\") pod \"isvc-tensorflow-predictor-88f6f6cb7-ntscd\" (UID: \"6305ba5b-257f-4df4-ba85-b9c2bb299358\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd"
Apr 16 17:12:38.189874 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:38.189841    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6305ba5b-257f-4df4-ba85-b9c2bb299358-kserve-provision-location\") pod \"isvc-tensorflow-predictor-88f6f6cb7-ntscd\" (UID: \"6305ba5b-257f-4df4-ba85-b9c2bb299358\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd"
Apr 16 17:12:38.190158 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:38.190142    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6305ba5b-257f-4df4-ba85-b9c2bb299358-kserve-provision-location\") pod \"isvc-tensorflow-predictor-88f6f6cb7-ntscd\" (UID: \"6305ba5b-257f-4df4-ba85-b9c2bb299358\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd"
Apr 16 17:12:38.338140 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:38.338064    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd"
Apr 16 17:12:38.454813 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:38.454783    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd"]
Apr 16 17:12:38.457736 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:12:38.457711    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6305ba5b_257f_4df4_ba85_b9c2bb299358.slice/crio-8d439660b12146480a390362f0f1a9034b4c6c89c92455a06956974e55e3077d WatchSource:0}: Error finding container 8d439660b12146480a390362f0f1a9034b4c6c89c92455a06956974e55e3077d: Status 404 returned error can't find the container with id 8d439660b12146480a390362f0f1a9034b4c6c89c92455a06956974e55e3077d
Apr 16 17:12:39.041426 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:39.041393    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd" event={"ID":"6305ba5b-257f-4df4-ba85-b9c2bb299358","Type":"ContainerStarted","Data":"8d569e921cee1941c26e6f8f48e99733972a8a81ea275d4bc14a69785e50062f"}
Apr 16 17:12:39.041426 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:39.041429    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd" event={"ID":"6305ba5b-257f-4df4-ba85-b9c2bb299358","Type":"ContainerStarted","Data":"8d439660b12146480a390362f0f1a9034b4c6c89c92455a06956974e55e3077d"}
Apr 16 17:12:41.619549 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:41.619528    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx"
Apr 16 17:12:41.716184 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:41.716156    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/847899df-39aa-4342-917e-ba98d9201dc0-kserve-provision-location\") pod \"847899df-39aa-4342-917e-ba98d9201dc0\" (UID: \"847899df-39aa-4342-917e-ba98d9201dc0\") "
Apr 16 17:12:41.716441 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:41.716421    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847899df-39aa-4342-917e-ba98d9201dc0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "847899df-39aa-4342-917e-ba98d9201dc0" (UID: "847899df-39aa-4342-917e-ba98d9201dc0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:12:41.819686 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:41.819660    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/847899df-39aa-4342-917e-ba98d9201dc0-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:12:42.052359 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:42.052284    2578 generic.go:358] "Generic (PLEG): container finished" podID="847899df-39aa-4342-917e-ba98d9201dc0" containerID="99615add3ddb639e08f1a0ff0af21d9e54697cdb8f7713f1c8396e8c312affa5" exitCode=0
Apr 16 17:12:42.052359 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:42.052348    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx" event={"ID":"847899df-39aa-4342-917e-ba98d9201dc0","Type":"ContainerDied","Data":"99615add3ddb639e08f1a0ff0af21d9e54697cdb8f7713f1c8396e8c312affa5"}
Apr 16 17:12:42.052575 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:42.052375    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx" event={"ID":"847899df-39aa-4342-917e-ba98d9201dc0","Type":"ContainerDied","Data":"7a6f1ca9c52bf485c7403cd3e7c24fb8fc77c356755c88d1c51b4511af9d2759"}
Apr 16 17:12:42.052575 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:42.052390    2578 scope.go:117] "RemoveContainer" containerID="99615add3ddb639e08f1a0ff0af21d9e54697cdb8f7713f1c8396e8c312affa5"
Apr 16 17:12:42.052575 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:42.052418    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx"
Apr 16 17:12:42.059769 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:42.059753    2578 scope.go:117] "RemoveContainer" containerID="bc0b9b7590bb1c8f638760894e23bdc0d7e217cdbbb18801fba3d84800421f50"
Apr 16 17:12:42.067193 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:42.067178    2578 scope.go:117] "RemoveContainer" containerID="99615add3ddb639e08f1a0ff0af21d9e54697cdb8f7713f1c8396e8c312affa5"
Apr 16 17:12:42.067439 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:12:42.067418    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99615add3ddb639e08f1a0ff0af21d9e54697cdb8f7713f1c8396e8c312affa5\": container with ID starting with 99615add3ddb639e08f1a0ff0af21d9e54697cdb8f7713f1c8396e8c312affa5 not found: ID does not exist" containerID="99615add3ddb639e08f1a0ff0af21d9e54697cdb8f7713f1c8396e8c312affa5"
Apr 16 17:12:42.067502 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:42.067447    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99615add3ddb639e08f1a0ff0af21d9e54697cdb8f7713f1c8396e8c312affa5"} err="failed to get container status \"99615add3ddb639e08f1a0ff0af21d9e54697cdb8f7713f1c8396e8c312affa5\": rpc error: code = NotFound desc = could not find container \"99615add3ddb639e08f1a0ff0af21d9e54697cdb8f7713f1c8396e8c312affa5\": container with ID starting with 99615add3ddb639e08f1a0ff0af21d9e54697cdb8f7713f1c8396e8c312affa5 not found: ID does not exist"
Apr 16 17:12:42.067502 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:42.067466    2578 scope.go:117] "RemoveContainer" containerID="bc0b9b7590bb1c8f638760894e23bdc0d7e217cdbbb18801fba3d84800421f50"
Apr 16 17:12:42.067748 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:12:42.067729    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0b9b7590bb1c8f638760894e23bdc0d7e217cdbbb18801fba3d84800421f50\": container with ID starting with bc0b9b7590bb1c8f638760894e23bdc0d7e217cdbbb18801fba3d84800421f50 not found: ID does not exist" containerID="bc0b9b7590bb1c8f638760894e23bdc0d7e217cdbbb18801fba3d84800421f50"
Apr 16 17:12:42.067820 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:42.067756    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0b9b7590bb1c8f638760894e23bdc0d7e217cdbbb18801fba3d84800421f50"} err="failed to get container status \"bc0b9b7590bb1c8f638760894e23bdc0d7e217cdbbb18801fba3d84800421f50\": rpc error: code = NotFound desc = could not find container \"bc0b9b7590bb1c8f638760894e23bdc0d7e217cdbbb18801fba3d84800421f50\": container with ID starting with bc0b9b7590bb1c8f638760894e23bdc0d7e217cdbbb18801fba3d84800421f50 not found: ID does not exist"
Apr 16 17:12:42.075485 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:42.075466    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx"]
Apr 16 17:12:42.079237 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:42.079217    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b446468df-dblzx"]
Apr 16 17:12:43.058172 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:43.058144    2578 generic.go:358] "Generic (PLEG): container finished" podID="6305ba5b-257f-4df4-ba85-b9c2bb299358" containerID="8d569e921cee1941c26e6f8f48e99733972a8a81ea275d4bc14a69785e50062f" exitCode=0
Apr 16 17:12:43.058482 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:43.058204    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd" event={"ID":"6305ba5b-257f-4df4-ba85-b9c2bb299358","Type":"ContainerDied","Data":"8d569e921cee1941c26e6f8f48e99733972a8a81ea275d4bc14a69785e50062f"}
Apr 16 17:12:43.235302 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:43.235277    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="847899df-39aa-4342-917e-ba98d9201dc0" path="/var/lib/kubelet/pods/847899df-39aa-4342-917e-ba98d9201dc0/volumes"
Apr 16 17:12:47.072147 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:47.072075    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd" event={"ID":"6305ba5b-257f-4df4-ba85-b9c2bb299358","Type":"ContainerStarted","Data":"e4ecad1cbdf65b0716fa6a0f752e1f4dbdba41cd5e427f27e001c3c4d39d57c7"}
Apr 16 17:12:47.072527 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:47.072346    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd"
Apr 16 17:12:47.073503 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:47.073477    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd" podUID="6305ba5b-257f-4df4-ba85-b9c2bb299358" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Apr 16 17:12:47.089382 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:47.089339    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd" podStartSLOduration=5.420114132 podStartE2EDuration="9.089327056s" podCreationTimestamp="2026-04-16 17:12:38 +0000 UTC" firstStartedPulling="2026-04-16 17:12:43.059242333 +0000 UTC m=+2572.456458699" lastFinishedPulling="2026-04-16 17:12:46.728455243 +0000 UTC m=+2576.125671623" observedRunningTime="2026-04-16 17:12:47.087486278 +0000 UTC m=+2576.484702665" watchObservedRunningTime="2026-04-16 17:12:47.089327056 +0000 UTC m=+2576.486543444"
Apr 16 17:12:48.075013 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:48.074974    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd" podUID="6305ba5b-257f-4df4-ba85-b9c2bb299358" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Apr 16 17:12:58.076409 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:12:58.076379    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd"
Apr 16 17:13:17.280784 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:17.280748    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd"]
Apr 16 17:13:17.281240 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:17.281008    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd" podUID="6305ba5b-257f-4df4-ba85-b9c2bb299358" containerName="kserve-container" containerID="cri-o://e4ecad1cbdf65b0716fa6a0f752e1f4dbdba41cd5e427f27e001c3c4d39d57c7" gracePeriod=30
Apr 16 17:13:17.334824 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:17.334795    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l"]
Apr 16 17:13:17.335104 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:17.335089    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847899df-39aa-4342-917e-ba98d9201dc0" containerName="storage-initializer"
Apr 16 17:13:17.335104 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:17.335102    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="847899df-39aa-4342-917e-ba98d9201dc0" containerName="storage-initializer"
Apr 16 17:13:17.335226 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:17.335112    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847899df-39aa-4342-917e-ba98d9201dc0" containerName="kserve-container"
Apr 16 17:13:17.335226 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:17.335117    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="847899df-39aa-4342-917e-ba98d9201dc0" containerName="kserve-container"
Apr 16 17:13:17.335226 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:17.335172    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="847899df-39aa-4342-917e-ba98d9201dc0" containerName="kserve-container"
Apr 16 17:13:17.337977 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:17.337956    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l"
Apr 16 17:13:17.346973 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:17.346947    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l"]
Apr 16 17:13:17.485604 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:17.485579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34bba70e-2cc8-4981-b43a-0b7520f2ef40-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-854bdff69c-cms7l\" (UID: \"34bba70e-2cc8-4981-b43a-0b7520f2ef40\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l"
Apr 16 17:13:17.586620 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:17.586542    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34bba70e-2cc8-4981-b43a-0b7520f2ef40-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-854bdff69c-cms7l\" (UID: \"34bba70e-2cc8-4981-b43a-0b7520f2ef40\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l"
Apr 16 17:13:17.586889 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:17.586872    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34bba70e-2cc8-4981-b43a-0b7520f2ef40-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-854bdff69c-cms7l\" (UID: \"34bba70e-2cc8-4981-b43a-0b7520f2ef40\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l"
Apr 16 17:13:17.648120 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:17.648095    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l"
Apr 16 17:13:17.763118 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:17.763094    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l"]
Apr 16 17:13:17.765784 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:13:17.765757    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34bba70e_2cc8_4981_b43a_0b7520f2ef40.slice/crio-00cc573ed619ee29c900ceae0d2a23e4d90b284197db03fed8e0773af47d46fe WatchSource:0}: Error finding container 00cc573ed619ee29c900ceae0d2a23e4d90b284197db03fed8e0773af47d46fe: Status 404 returned error can't find the container with id 00cc573ed619ee29c900ceae0d2a23e4d90b284197db03fed8e0773af47d46fe
Apr 16 17:13:18.167898 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:18.167858    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l" event={"ID":"34bba70e-2cc8-4981-b43a-0b7520f2ef40","Type":"ContainerStarted","Data":"8a106027c6cf36e131e55b93e5fec6bda67a0e445e3b1ef5a19e133a0ebc3d0b"}
Apr 16 17:13:18.167898 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:18.167897    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l" event={"ID":"34bba70e-2cc8-4981-b43a-0b7520f2ef40","Type":"ContainerStarted","Data":"00cc573ed619ee29c900ceae0d2a23e4d90b284197db03fed8e0773af47d46fe"}
Apr 16 17:13:23.183897 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:23.183865    2578 generic.go:358] "Generic (PLEG): container finished" podID="34bba70e-2cc8-4981-b43a-0b7520f2ef40" containerID="8a106027c6cf36e131e55b93e5fec6bda67a0e445e3b1ef5a19e133a0ebc3d0b" exitCode=0
Apr 16 17:13:23.184258 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:23.183918    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l" event={"ID":"34bba70e-2cc8-4981-b43a-0b7520f2ef40","Type":"ContainerDied","Data":"8a106027c6cf36e131e55b93e5fec6bda67a0e445e3b1ef5a19e133a0ebc3d0b"}
Apr 16 17:13:24.187670 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:24.187626    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l" event={"ID":"34bba70e-2cc8-4981-b43a-0b7520f2ef40","Type":"ContainerStarted","Data":"43fc38e195cdc954765a7e46207442be5ae00146c7957eb8cd60d02361997dbf"}
Apr 16 17:13:24.188057 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:24.187912    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l"
Apr 16 17:13:24.189210 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:24.189183    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l" podUID="34bba70e-2cc8-4981-b43a-0b7520f2ef40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Apr 16 17:13:24.203664 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:24.203619    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l" podStartSLOduration=7.203606218 podStartE2EDuration="7.203606218s" podCreationTimestamp="2026-04-16 17:13:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:13:24.202483719 +0000 UTC m=+2613.599700107" watchObservedRunningTime="2026-04-16 17:13:24.203606218 +0000 UTC m=+2613.600822607"
Apr 16 17:13:25.191326 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:25.191284    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l" podUID="34bba70e-2cc8-4981-b43a-0b7520f2ef40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Apr 16 17:13:35.192428 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:35.192394    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l"
Apr 16 17:13:46.476978 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:46.476937    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l"]
Apr 16 17:13:46.477459 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:46.477206    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l" podUID="34bba70e-2cc8-4981-b43a-0b7520f2ef40" containerName="kserve-container" containerID="cri-o://43fc38e195cdc954765a7e46207442be5ae00146c7957eb8cd60d02361997dbf" gracePeriod=30
Apr 16 17:13:46.530111 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:46.530078    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb"]
Apr 16 17:13:46.533433 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:46.533416    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb"
Apr 16 17:13:46.544320 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:46.544298    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb"]
Apr 16 17:13:46.691103 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:46.691068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/381b8e76-3644-43ba-b121-7bfd3d0ac6b4-kserve-provision-location\") pod \"isvc-triton-predictor-85f9f46646-vvdhb\" (UID: \"381b8e76-3644-43ba-b121-7bfd3d0ac6b4\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb"
Apr 16 17:13:46.791640 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:46.791544    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/381b8e76-3644-43ba-b121-7bfd3d0ac6b4-kserve-provision-location\") pod \"isvc-triton-predictor-85f9f46646-vvdhb\" (UID: \"381b8e76-3644-43ba-b121-7bfd3d0ac6b4\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb"
Apr 16 17:13:46.791899 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:46.791879    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/381b8e76-3644-43ba-b121-7bfd3d0ac6b4-kserve-provision-location\") pod \"isvc-triton-predictor-85f9f46646-vvdhb\" (UID: \"381b8e76-3644-43ba-b121-7bfd3d0ac6b4\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb"
Apr 16 17:13:46.843632 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:46.843610    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb"
Apr 16 17:13:46.960249 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:46.960226    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb"]
Apr 16 17:13:46.962230 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:13:46.962185    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381b8e76_3644_43ba_b121_7bfd3d0ac6b4.slice/crio-dad6dc4436aebb99c8df088f519f35412c05d4c6bb82ee3ed5014c438a8f19a9 WatchSource:0}: Error finding container dad6dc4436aebb99c8df088f519f35412c05d4c6bb82ee3ed5014c438a8f19a9: Status 404 returned error can't find the container with id dad6dc4436aebb99c8df088f519f35412c05d4c6bb82ee3ed5014c438a8f19a9
Apr 16 17:13:47.261030 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:47.260992    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb" event={"ID":"381b8e76-3644-43ba-b121-7bfd3d0ac6b4","Type":"ContainerStarted","Data":"b686bba5601fd044cd4a4bb343ea7fe605b191a914f8bdd978629f32a4bd5cb8"}
Apr 16 17:13:47.261030 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:47.261029    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb" event={"ID":"381b8e76-3644-43ba-b121-7bfd3d0ac6b4","Type":"ContainerStarted","Data":"dad6dc4436aebb99c8df088f519f35412c05d4c6bb82ee3ed5014c438a8f19a9"}
Apr 16 17:13:47.917191 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:47.917162    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd"
Apr 16 17:13:48.002409 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:48.002382    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6305ba5b-257f-4df4-ba85-b9c2bb299358-kserve-provision-location\") pod \"6305ba5b-257f-4df4-ba85-b9c2bb299358\" (UID: \"6305ba5b-257f-4df4-ba85-b9c2bb299358\") "
Apr 16 17:13:48.012844 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:48.012817    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6305ba5b-257f-4df4-ba85-b9c2bb299358-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6305ba5b-257f-4df4-ba85-b9c2bb299358" (UID: "6305ba5b-257f-4df4-ba85-b9c2bb299358"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:13:48.103774 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:48.103716    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6305ba5b-257f-4df4-ba85-b9c2bb299358-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:13:48.265607 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:48.265582    2578 generic.go:358] "Generic (PLEG): container finished" podID="6305ba5b-257f-4df4-ba85-b9c2bb299358" containerID="e4ecad1cbdf65b0716fa6a0f752e1f4dbdba41cd5e427f27e001c3c4d39d57c7" exitCode=137
Apr 16 17:13:48.265704 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:48.265640    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd"
Apr 16 17:13:48.265704 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:48.265665    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd" event={"ID":"6305ba5b-257f-4df4-ba85-b9c2bb299358","Type":"ContainerDied","Data":"e4ecad1cbdf65b0716fa6a0f752e1f4dbdba41cd5e427f27e001c3c4d39d57c7"}
Apr 16 17:13:48.265704 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:48.265701    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd" event={"ID":"6305ba5b-257f-4df4-ba85-b9c2bb299358","Type":"ContainerDied","Data":"8d439660b12146480a390362f0f1a9034b4c6c89c92455a06956974e55e3077d"}
Apr 16 17:13:48.265840 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:48.265716    2578 scope.go:117] "RemoveContainer" containerID="e4ecad1cbdf65b0716fa6a0f752e1f4dbdba41cd5e427f27e001c3c4d39d57c7"
Apr 16 17:13:48.273689 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:48.273667    2578 scope.go:117] "RemoveContainer" containerID="8d569e921cee1941c26e6f8f48e99733972a8a81ea275d4bc14a69785e50062f"
Apr 16 17:13:48.281413 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:48.281396    2578 scope.go:117] "RemoveContainer" containerID="e4ecad1cbdf65b0716fa6a0f752e1f4dbdba41cd5e427f27e001c3c4d39d57c7"
Apr 16 17:13:48.281679 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:13:48.281662    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ecad1cbdf65b0716fa6a0f752e1f4dbdba41cd5e427f27e001c3c4d39d57c7\": container with ID starting with e4ecad1cbdf65b0716fa6a0f752e1f4dbdba41cd5e427f27e001c3c4d39d57c7 not found: ID does not exist" containerID="e4ecad1cbdf65b0716fa6a0f752e1f4dbdba41cd5e427f27e001c3c4d39d57c7"
Apr 16 17:13:48.281735 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:48.281688    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ecad1cbdf65b0716fa6a0f752e1f4dbdba41cd5e427f27e001c3c4d39d57c7"} err="failed to get container status \"e4ecad1cbdf65b0716fa6a0f752e1f4dbdba41cd5e427f27e001c3c4d39d57c7\": rpc error: code = NotFound desc = could not find container \"e4ecad1cbdf65b0716fa6a0f752e1f4dbdba41cd5e427f27e001c3c4d39d57c7\": container with ID starting with e4ecad1cbdf65b0716fa6a0f752e1f4dbdba41cd5e427f27e001c3c4d39d57c7 not found: ID does not exist"
Apr 16 17:13:48.281735 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:48.281706    2578 scope.go:117] "RemoveContainer" containerID="8d569e921cee1941c26e6f8f48e99733972a8a81ea275d4bc14a69785e50062f"
Apr 16 17:13:48.281909 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:13:48.281895    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d569e921cee1941c26e6f8f48e99733972a8a81ea275d4bc14a69785e50062f\": container with ID starting with 8d569e921cee1941c26e6f8f48e99733972a8a81ea275d4bc14a69785e50062f not found: ID does not exist" containerID="8d569e921cee1941c26e6f8f48e99733972a8a81ea275d4bc14a69785e50062f"
Apr 16 17:13:48.281949 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:48.281912    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d569e921cee1941c26e6f8f48e99733972a8a81ea275d4bc14a69785e50062f"} err="failed to get container status \"8d569e921cee1941c26e6f8f48e99733972a8a81ea275d4bc14a69785e50062f\": rpc error: code = NotFound desc = could not find container \"8d569e921cee1941c26e6f8f48e99733972a8a81ea275d4bc14a69785e50062f\": container with ID starting with 8d569e921cee1941c26e6f8f48e99733972a8a81ea275d4bc14a69785e50062f not found: ID does not exist"
Apr 16 17:13:48.285476 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:48.285458    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd"]
Apr 16 17:13:48.289821 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:48.289801    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-ntscd"]
Apr 16 17:13:49.235113 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:49.235082    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6305ba5b-257f-4df4-ba85-b9c2bb299358" path="/var/lib/kubelet/pods/6305ba5b-257f-4df4-ba85-b9c2bb299358/volumes"
Apr 16 17:13:51.279316 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:51.279252    2578 generic.go:358] "Generic (PLEG): container finished" podID="381b8e76-3644-43ba-b121-7bfd3d0ac6b4" containerID="b686bba5601fd044cd4a4bb343ea7fe605b191a914f8bdd978629f32a4bd5cb8" exitCode=0
Apr 16 17:13:51.279633 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:13:51.279323    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb" event={"ID":"381b8e76-3644-43ba-b121-7bfd3d0ac6b4","Type":"ContainerDied","Data":"b686bba5601fd044cd4a4bb343ea7fe605b191a914f8bdd978629f32a4bd5cb8"}
Apr 16 17:14:17.270793 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:17.270728    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l"
Apr 16 17:14:17.365407 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:17.365367    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34bba70e-2cc8-4981-b43a-0b7520f2ef40-kserve-provision-location\") pod \"34bba70e-2cc8-4981-b43a-0b7520f2ef40\" (UID: \"34bba70e-2cc8-4981-b43a-0b7520f2ef40\") "
Apr 16 17:14:17.370272 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:17.370239    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34bba70e-2cc8-4981-b43a-0b7520f2ef40-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34bba70e-2cc8-4981-b43a-0b7520f2ef40" (UID: "34bba70e-2cc8-4981-b43a-0b7520f2ef40"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:14:17.388669 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:17.388523    2578 generic.go:358] "Generic (PLEG): container finished" podID="34bba70e-2cc8-4981-b43a-0b7520f2ef40" containerID="43fc38e195cdc954765a7e46207442be5ae00146c7957eb8cd60d02361997dbf" exitCode=137
Apr 16 17:14:17.388669 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:17.388581    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l" event={"ID":"34bba70e-2cc8-4981-b43a-0b7520f2ef40","Type":"ContainerDied","Data":"43fc38e195cdc954765a7e46207442be5ae00146c7957eb8cd60d02361997dbf"}
Apr 16 17:14:17.388669 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:17.388628    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l" event={"ID":"34bba70e-2cc8-4981-b43a-0b7520f2ef40","Type":"ContainerDied","Data":"00cc573ed619ee29c900ceae0d2a23e4d90b284197db03fed8e0773af47d46fe"}
Apr 16 17:14:17.388669 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:17.388634    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l"
Apr 16 17:14:17.388669 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:17.388649    2578 scope.go:117] "RemoveContainer" containerID="43fc38e195cdc954765a7e46207442be5ae00146c7957eb8cd60d02361997dbf"
Apr 16 17:14:17.400762 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:17.400405    2578 scope.go:117] "RemoveContainer" containerID="8a106027c6cf36e131e55b93e5fec6bda67a0e445e3b1ef5a19e133a0ebc3d0b"
Apr 16 17:14:17.412366 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:17.412248    2578 scope.go:117] "RemoveContainer" containerID="43fc38e195cdc954765a7e46207442be5ae00146c7957eb8cd60d02361997dbf"
Apr 16 17:14:17.412977 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:14:17.412947    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43fc38e195cdc954765a7e46207442be5ae00146c7957eb8cd60d02361997dbf\": container with ID starting with 43fc38e195cdc954765a7e46207442be5ae00146c7957eb8cd60d02361997dbf not found: ID does not exist" containerID="43fc38e195cdc954765a7e46207442be5ae00146c7957eb8cd60d02361997dbf"
Apr 16 17:14:17.413081 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:17.412989    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43fc38e195cdc954765a7e46207442be5ae00146c7957eb8cd60d02361997dbf"} err="failed to get container status \"43fc38e195cdc954765a7e46207442be5ae00146c7957eb8cd60d02361997dbf\": rpc error: code = NotFound desc = could not find container \"43fc38e195cdc954765a7e46207442be5ae00146c7957eb8cd60d02361997dbf\": container with ID starting with 43fc38e195cdc954765a7e46207442be5ae00146c7957eb8cd60d02361997dbf not found: ID does not exist"
Apr 16 17:14:17.413081 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:17.413014    2578 scope.go:117] "RemoveContainer" containerID="8a106027c6cf36e131e55b93e5fec6bda67a0e445e3b1ef5a19e133a0ebc3d0b"
Apr 16 17:14:17.413431 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:14:17.413408    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a106027c6cf36e131e55b93e5fec6bda67a0e445e3b1ef5a19e133a0ebc3d0b\": container with ID starting with 8a106027c6cf36e131e55b93e5fec6bda67a0e445e3b1ef5a19e133a0ebc3d0b not found: ID does not exist" containerID="8a106027c6cf36e131e55b93e5fec6bda67a0e445e3b1ef5a19e133a0ebc3d0b"
Apr 16 17:14:17.413514 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:17.413440    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a106027c6cf36e131e55b93e5fec6bda67a0e445e3b1ef5a19e133a0ebc3d0b"} err="failed to get container status \"8a106027c6cf36e131e55b93e5fec6bda67a0e445e3b1ef5a19e133a0ebc3d0b\": rpc error: code = NotFound desc = could not find container \"8a106027c6cf36e131e55b93e5fec6bda67a0e445e3b1ef5a19e133a0ebc3d0b\": container with ID starting with 8a106027c6cf36e131e55b93e5fec6bda67a0e445e3b1ef5a19e133a0ebc3d0b not found: ID does not exist"
Apr 16 17:14:17.416009 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:17.415968    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l"]
Apr 16 17:14:17.417330 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:17.417304    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-cms7l"]
Apr 16 17:14:17.467030 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:17.467003    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34bba70e-2cc8-4981-b43a-0b7520f2ef40-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:14:19.236509 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:14:19.236118    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34bba70e-2cc8-4981-b43a-0b7520f2ef40" path="/var/lib/kubelet/pods/34bba70e-2cc8-4981-b43a-0b7520f2ef40/volumes"
Apr 16 17:15:44.699841 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:15:44.699809    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 17:15:44.700372 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:15:44.699809    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 17:15:46.681320 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:15:46.681289    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb" event={"ID":"381b8e76-3644-43ba-b121-7bfd3d0ac6b4","Type":"ContainerStarted","Data":"2f7d4fbe0d7c89e8e387905664e2c22afd04ef95dc9cec322c50e96d6c4b3f36"}
Apr 16 17:15:46.681725 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:15:46.681491    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb"
Apr 16 17:15:46.682744 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:15:46.682710    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb" podUID="381b8e76-3644-43ba-b121-7bfd3d0ac6b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Apr 16 17:15:46.697173 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:15:46.697133    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb" podStartSLOduration=6.290168801 podStartE2EDuration="2m0.697121083s" podCreationTimestamp="2026-04-16 17:13:46 +0000 UTC" firstStartedPulling="2026-04-16 17:13:51.280339306 +0000 UTC m=+2640.677555673" lastFinishedPulling="2026-04-16 17:15:45.687291574 +0000 UTC m=+2755.084507955" observedRunningTime="2026-04-16 17:15:46.69587339 +0000 UTC m=+2756.093089802" watchObservedRunningTime="2026-04-16 17:15:46.697121083 +0000 UTC m=+2756.094337471"
Apr 16 17:15:47.684202 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:15:47.684166    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb" podUID="381b8e76-3644-43ba-b121-7bfd3d0ac6b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Apr 16 17:15:57.685329 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:15:57.685295    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb"
Apr 16 17:16:08.101072 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.100994    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb"]
Apr 16 17:16:08.101468 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.101229    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb" podUID="381b8e76-3644-43ba-b121-7bfd3d0ac6b4" containerName="kserve-container" containerID="cri-o://2f7d4fbe0d7c89e8e387905664e2c22afd04ef95dc9cec322c50e96d6c4b3f36" gracePeriod=30
Apr 16 17:16:08.183122 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.183091    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw"]
Apr 16 17:16:08.183413 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.183400    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34bba70e-2cc8-4981-b43a-0b7520f2ef40" containerName="kserve-container"
Apr 16 17:16:08.183459 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.183415    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34bba70e-2cc8-4981-b43a-0b7520f2ef40" containerName="kserve-container"
Apr 16 17:16:08.183459 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.183424    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6305ba5b-257f-4df4-ba85-b9c2bb299358" containerName="storage-initializer"
Apr 16 17:16:08.183459 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.183429    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6305ba5b-257f-4df4-ba85-b9c2bb299358" containerName="storage-initializer"
Apr 16 17:16:08.183459 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.183441    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34bba70e-2cc8-4981-b43a-0b7520f2ef40" containerName="storage-initializer"
Apr 16 17:16:08.183459 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.183446    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34bba70e-2cc8-4981-b43a-0b7520f2ef40" containerName="storage-initializer"
Apr 16 17:16:08.183459 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.183458    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6305ba5b-257f-4df4-ba85-b9c2bb299358" containerName="kserve-container"
Apr 16 17:16:08.183675 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.183463    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6305ba5b-257f-4df4-ba85-b9c2bb299358" containerName="kserve-container"
Apr 16 17:16:08.183675 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.183506    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="6305ba5b-257f-4df4-ba85-b9c2bb299358" containerName="kserve-container"
Apr 16 17:16:08.183675 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.183518    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="34bba70e-2cc8-4981-b43a-0b7520f2ef40" containerName="kserve-container"
Apr 16 17:16:08.188478 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.188456    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw"
Apr 16 17:16:08.194510 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.194489    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw"]
Apr 16 17:16:08.273618 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.273592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f6ecbab-a599-47f8-adbc-4ed01883b9a4-kserve-provision-location\") pod \"isvc-xgboost-predictor-6dbc9d6d47-lh8dw\" (UID: \"9f6ecbab-a599-47f8-adbc-4ed01883b9a4\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw"
Apr 16 17:16:08.374896 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.374814    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f6ecbab-a599-47f8-adbc-4ed01883b9a4-kserve-provision-location\") pod \"isvc-xgboost-predictor-6dbc9d6d47-lh8dw\" (UID: \"9f6ecbab-a599-47f8-adbc-4ed01883b9a4\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw"
Apr 16 17:16:08.375163 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.375143    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f6ecbab-a599-47f8-adbc-4ed01883b9a4-kserve-provision-location\") pod \"isvc-xgboost-predictor-6dbc9d6d47-lh8dw\" (UID: \"9f6ecbab-a599-47f8-adbc-4ed01883b9a4\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw"
Apr 16 17:16:08.500058 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.500034    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw"
Apr 16 17:16:08.678679 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.678656    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw"]
Apr 16 17:16:08.681170 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:16:08.681141    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6ecbab_a599_47f8_adbc_4ed01883b9a4.slice/crio-29bbab104afd0c3e1ba098a410191f3b253a945c1d7ecd724649393a05bc4181 WatchSource:0}: Error finding container 29bbab104afd0c3e1ba098a410191f3b253a945c1d7ecd724649393a05bc4181: Status 404 returned error can't find the container with id 29bbab104afd0c3e1ba098a410191f3b253a945c1d7ecd724649393a05bc4181
Apr 16 17:16:08.682951 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.682935    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 17:16:08.741270 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.741238    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw" event={"ID":"9f6ecbab-a599-47f8-adbc-4ed01883b9a4","Type":"ContainerStarted","Data":"5fd69affb622984ce7be4f827e482849e9b08a77f8753c60eaadd02360ec14b1"}
Apr 16 17:16:08.741270 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:08.741269    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw" event={"ID":"9f6ecbab-a599-47f8-adbc-4ed01883b9a4","Type":"ContainerStarted","Data":"29bbab104afd0c3e1ba098a410191f3b253a945c1d7ecd724649393a05bc4181"}
Apr 16 17:16:10.233141 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:10.233123    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb"
Apr 16 17:16:10.290333 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:10.290263    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/381b8e76-3644-43ba-b121-7bfd3d0ac6b4-kserve-provision-location\") pod \"381b8e76-3644-43ba-b121-7bfd3d0ac6b4\" (UID: \"381b8e76-3644-43ba-b121-7bfd3d0ac6b4\") "
Apr 16 17:16:10.290679 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:10.290656    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381b8e76-3644-43ba-b121-7bfd3d0ac6b4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "381b8e76-3644-43ba-b121-7bfd3d0ac6b4" (UID: "381b8e76-3644-43ba-b121-7bfd3d0ac6b4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:16:10.391238 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:10.391215    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/381b8e76-3644-43ba-b121-7bfd3d0ac6b4-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:16:10.748167 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:10.748139    2578 generic.go:358] "Generic (PLEG): container finished" podID="381b8e76-3644-43ba-b121-7bfd3d0ac6b4" containerID="2f7d4fbe0d7c89e8e387905664e2c22afd04ef95dc9cec322c50e96d6c4b3f36" exitCode=0
Apr 16 17:16:10.748283 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:10.748214    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb"
Apr 16 17:16:10.748283 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:10.748230    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb" event={"ID":"381b8e76-3644-43ba-b121-7bfd3d0ac6b4","Type":"ContainerDied","Data":"2f7d4fbe0d7c89e8e387905664e2c22afd04ef95dc9cec322c50e96d6c4b3f36"}
Apr 16 17:16:10.748283 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:10.748271    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb" event={"ID":"381b8e76-3644-43ba-b121-7bfd3d0ac6b4","Type":"ContainerDied","Data":"dad6dc4436aebb99c8df088f519f35412c05d4c6bb82ee3ed5014c438a8f19a9"}
Apr 16 17:16:10.748420 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:10.748288    2578 scope.go:117] "RemoveContainer" containerID="2f7d4fbe0d7c89e8e387905664e2c22afd04ef95dc9cec322c50e96d6c4b3f36"
Apr 16 17:16:10.756399 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:10.756364    2578 scope.go:117] "RemoveContainer" containerID="b686bba5601fd044cd4a4bb343ea7fe605b191a914f8bdd978629f32a4bd5cb8"
Apr 16 17:16:10.764976 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:10.764952    2578 scope.go:117] "RemoveContainer" containerID="2f7d4fbe0d7c89e8e387905664e2c22afd04ef95dc9cec322c50e96d6c4b3f36"
Apr 16 17:16:10.765232 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:16:10.765214    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f7d4fbe0d7c89e8e387905664e2c22afd04ef95dc9cec322c50e96d6c4b3f36\": container with ID starting with 2f7d4fbe0d7c89e8e387905664e2c22afd04ef95dc9cec322c50e96d6c4b3f36 not found: ID does not exist" containerID="2f7d4fbe0d7c89e8e387905664e2c22afd04ef95dc9cec322c50e96d6c4b3f36"
Apr 16 17:16:10.765296 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:10.765239    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f7d4fbe0d7c89e8e387905664e2c22afd04ef95dc9cec322c50e96d6c4b3f36"} err="failed to get container status \"2f7d4fbe0d7c89e8e387905664e2c22afd04ef95dc9cec322c50e96d6c4b3f36\": rpc error: code = NotFound desc = could not find container \"2f7d4fbe0d7c89e8e387905664e2c22afd04ef95dc9cec322c50e96d6c4b3f36\": container with ID starting with 2f7d4fbe0d7c89e8e387905664e2c22afd04ef95dc9cec322c50e96d6c4b3f36 not found: ID does not exist"
Apr 16 17:16:10.765296 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:10.765258    2578 scope.go:117] "RemoveContainer" containerID="b686bba5601fd044cd4a4bb343ea7fe605b191a914f8bdd978629f32a4bd5cb8"
Apr 16 17:16:10.765489 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:16:10.765472    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b686bba5601fd044cd4a4bb343ea7fe605b191a914f8bdd978629f32a4bd5cb8\": container with ID starting with b686bba5601fd044cd4a4bb343ea7fe605b191a914f8bdd978629f32a4bd5cb8 not found: ID does not exist" containerID="b686bba5601fd044cd4a4bb343ea7fe605b191a914f8bdd978629f32a4bd5cb8"
Apr 16 17:16:10.765530 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:10.765497    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b686bba5601fd044cd4a4bb343ea7fe605b191a914f8bdd978629f32a4bd5cb8"} err="failed to get container status \"b686bba5601fd044cd4a4bb343ea7fe605b191a914f8bdd978629f32a4bd5cb8\": rpc error: code = NotFound desc = could not find container \"b686bba5601fd044cd4a4bb343ea7fe605b191a914f8bdd978629f32a4bd5cb8\": container with ID starting with b686bba5601fd044cd4a4bb343ea7fe605b191a914f8bdd978629f32a4bd5cb8 not found: ID does not exist"
Apr 16 17:16:10.768684 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:10.768663    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb"]
Apr 16 17:16:10.772351 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:10.772331    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-vvdhb"]
Apr 16 17:16:11.239420 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:11.239389    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="381b8e76-3644-43ba-b121-7bfd3d0ac6b4" path="/var/lib/kubelet/pods/381b8e76-3644-43ba-b121-7bfd3d0ac6b4/volumes"
Apr 16 17:16:13.758477 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:13.758450    2578 generic.go:358] "Generic (PLEG): container finished" podID="9f6ecbab-a599-47f8-adbc-4ed01883b9a4" containerID="5fd69affb622984ce7be4f827e482849e9b08a77f8753c60eaadd02360ec14b1" exitCode=0
Apr 16 17:16:13.758953 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:13.758529    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw" event={"ID":"9f6ecbab-a599-47f8-adbc-4ed01883b9a4","Type":"ContainerDied","Data":"5fd69affb622984ce7be4f827e482849e9b08a77f8753c60eaadd02360ec14b1"}
Apr 16 17:16:33.825274 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:33.825247    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw" event={"ID":"9f6ecbab-a599-47f8-adbc-4ed01883b9a4","Type":"ContainerStarted","Data":"a9112ef69918baad73cb0f5c2a7750a56557c1322db399df207c7a33acfcc175"}
Apr 16 17:16:33.825613 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:33.825512    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw"
Apr 16 17:16:33.826677 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:33.826651    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw" podUID="9f6ecbab-a599-47f8-adbc-4ed01883b9a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Apr 16 17:16:33.841601 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:33.841515    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw" podStartSLOduration=5.900552138 podStartE2EDuration="25.841504455s" podCreationTimestamp="2026-04-16 17:16:08 +0000 UTC" firstStartedPulling="2026-04-16 17:16:13.759635997 +0000 UTC m=+2783.156852364" lastFinishedPulling="2026-04-16 17:16:33.700588316 +0000 UTC m=+2803.097804681" observedRunningTime="2026-04-16 17:16:33.839338807 +0000 UTC m=+2803.236555196" watchObservedRunningTime="2026-04-16 17:16:33.841504455 +0000 UTC m=+2803.238720843"
Apr 16 17:16:34.829452 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:34.829418    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw" podUID="9f6ecbab-a599-47f8-adbc-4ed01883b9a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Apr 16 17:16:44.830121 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:44.830079    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw" podUID="9f6ecbab-a599-47f8-adbc-4ed01883b9a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Apr 16 17:16:54.830338 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:16:54.830300    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw" podUID="9f6ecbab-a599-47f8-adbc-4ed01883b9a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Apr 16 17:17:04.829720 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:04.829673    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw" podUID="9f6ecbab-a599-47f8-adbc-4ed01883b9a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Apr 16 17:17:14.830079 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:14.830035    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw" podUID="9f6ecbab-a599-47f8-adbc-4ed01883b9a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Apr 16 17:17:24.829352 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:24.829304    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw" podUID="9f6ecbab-a599-47f8-adbc-4ed01883b9a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Apr 16 17:17:34.831321 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:34.831237    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw"
Apr 16 17:17:38.333247 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:38.333218    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw"]
Apr 16 17:17:38.333625 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:38.333458    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw" podUID="9f6ecbab-a599-47f8-adbc-4ed01883b9a4" containerName="kserve-container" containerID="cri-o://a9112ef69918baad73cb0f5c2a7750a56557c1322db399df207c7a33acfcc175" gracePeriod=30
Apr 16 17:17:38.438611 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:38.438586    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5"]
Apr 16 17:17:38.438884 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:38.438873    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381b8e76-3644-43ba-b121-7bfd3d0ac6b4" containerName="kserve-container"
Apr 16 17:17:38.438929 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:38.438886    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="381b8e76-3644-43ba-b121-7bfd3d0ac6b4" containerName="kserve-container"
Apr 16 17:17:38.438929 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:38.438909    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381b8e76-3644-43ba-b121-7bfd3d0ac6b4" containerName="storage-initializer"
Apr 16 17:17:38.438929 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:38.438914    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="381b8e76-3644-43ba-b121-7bfd3d0ac6b4" containerName="storage-initializer"
Apr 16 17:17:38.439025 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:38.438973    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="381b8e76-3644-43ba-b121-7bfd3d0ac6b4" containerName="kserve-container"
Apr 16 17:17:38.442017 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:38.442000    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5"
Apr 16 17:17:38.451500 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:38.451474    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5"]
Apr 16 17:17:38.558096 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:38.558065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e5a51cf-608e-4c37-9b18-9fbfdd71494b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5\" (UID: \"5e5a51cf-608e-4c37-9b18-9fbfdd71494b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5"
Apr 16 17:17:38.658471 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:38.658412    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e5a51cf-608e-4c37-9b18-9fbfdd71494b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5\" (UID: \"5e5a51cf-608e-4c37-9b18-9fbfdd71494b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5"
Apr 16 17:17:38.658781 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:38.658761    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e5a51cf-608e-4c37-9b18-9fbfdd71494b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5\" (UID: \"5e5a51cf-608e-4c37-9b18-9fbfdd71494b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5"
Apr 16 17:17:38.752967 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:38.752943    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5"
Apr 16 17:17:38.866366 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:38.866344    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5"]
Apr 16 17:17:38.868791 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:17:38.868758    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5a51cf_608e_4c37_9b18_9fbfdd71494b.slice/crio-dd2043f74f78068a1157514b9ba5d3d8f7afa84389be0cbccfc757bc9cc25739 WatchSource:0}: Error finding container dd2043f74f78068a1157514b9ba5d3d8f7afa84389be0cbccfc757bc9cc25739: Status 404 returned error can't find the container with id dd2043f74f78068a1157514b9ba5d3d8f7afa84389be0cbccfc757bc9cc25739
Apr 16 17:17:39.022455 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:39.022423    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5" event={"ID":"5e5a51cf-608e-4c37-9b18-9fbfdd71494b","Type":"ContainerStarted","Data":"26577f15c89889fcc685efaadac7595b034da4e776f3b20aac3d7449c17390dd"}
Apr 16 17:17:39.022455 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:39.022457    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5" event={"ID":"5e5a51cf-608e-4c37-9b18-9fbfdd71494b","Type":"ContainerStarted","Data":"dd2043f74f78068a1157514b9ba5d3d8f7afa84389be0cbccfc757bc9cc25739"}
Apr 16 17:17:41.366418 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:41.366395    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw"
Apr 16 17:17:41.479237 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:41.479171    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f6ecbab-a599-47f8-adbc-4ed01883b9a4-kserve-provision-location\") pod \"9f6ecbab-a599-47f8-adbc-4ed01883b9a4\" (UID: \"9f6ecbab-a599-47f8-adbc-4ed01883b9a4\") "
Apr 16 17:17:41.479446 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:41.479421    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6ecbab-a599-47f8-adbc-4ed01883b9a4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f6ecbab-a599-47f8-adbc-4ed01883b9a4" (UID: "9f6ecbab-a599-47f8-adbc-4ed01883b9a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:17:41.580269 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:41.580243    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f6ecbab-a599-47f8-adbc-4ed01883b9a4-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:17:42.033142 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:42.033116    2578 generic.go:358] "Generic (PLEG): container finished" podID="9f6ecbab-a599-47f8-adbc-4ed01883b9a4" containerID="a9112ef69918baad73cb0f5c2a7750a56557c1322db399df207c7a33acfcc175" exitCode=0
Apr 16 17:17:42.033299 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:42.033164    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw" event={"ID":"9f6ecbab-a599-47f8-adbc-4ed01883b9a4","Type":"ContainerDied","Data":"a9112ef69918baad73cb0f5c2a7750a56557c1322db399df207c7a33acfcc175"}
Apr 16 17:17:42.033299 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:42.033178    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw"
Apr 16 17:17:42.033299 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:42.033187    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw" event={"ID":"9f6ecbab-a599-47f8-adbc-4ed01883b9a4","Type":"ContainerDied","Data":"29bbab104afd0c3e1ba098a410191f3b253a945c1d7ecd724649393a05bc4181"}
Apr 16 17:17:42.033299 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:42.033202    2578 scope.go:117] "RemoveContainer" containerID="a9112ef69918baad73cb0f5c2a7750a56557c1322db399df207c7a33acfcc175"
Apr 16 17:17:42.040365 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:42.040345    2578 scope.go:117] "RemoveContainer" containerID="5fd69affb622984ce7be4f827e482849e9b08a77f8753c60eaadd02360ec14b1"
Apr 16 17:17:42.047255 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:42.047238    2578 scope.go:117] "RemoveContainer" containerID="a9112ef69918baad73cb0f5c2a7750a56557c1322db399df207c7a33acfcc175"
Apr 16 17:17:42.047497 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:17:42.047480    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9112ef69918baad73cb0f5c2a7750a56557c1322db399df207c7a33acfcc175\": container with ID starting with a9112ef69918baad73cb0f5c2a7750a56557c1322db399df207c7a33acfcc175 not found: ID does not exist" containerID="a9112ef69918baad73cb0f5c2a7750a56557c1322db399df207c7a33acfcc175"
Apr 16 17:17:42.047541 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:42.047504    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9112ef69918baad73cb0f5c2a7750a56557c1322db399df207c7a33acfcc175"} err="failed to get container status \"a9112ef69918baad73cb0f5c2a7750a56557c1322db399df207c7a33acfcc175\": rpc error: code = NotFound desc = could not find container \"a9112ef69918baad73cb0f5c2a7750a56557c1322db399df207c7a33acfcc175\": container with ID starting with a9112ef69918baad73cb0f5c2a7750a56557c1322db399df207c7a33acfcc175 not found: ID does not exist"
Apr 16 17:17:42.047541 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:42.047520    2578 scope.go:117] "RemoveContainer" containerID="5fd69affb622984ce7be4f827e482849e9b08a77f8753c60eaadd02360ec14b1"
Apr 16 17:17:42.047725 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:17:42.047709    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd69affb622984ce7be4f827e482849e9b08a77f8753c60eaadd02360ec14b1\": container with ID starting with 5fd69affb622984ce7be4f827e482849e9b08a77f8753c60eaadd02360ec14b1 not found: ID does not exist" containerID="5fd69affb622984ce7be4f827e482849e9b08a77f8753c60eaadd02360ec14b1"
Apr 16 17:17:42.047773 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:42.047729    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd69affb622984ce7be4f827e482849e9b08a77f8753c60eaadd02360ec14b1"} err="failed to get container status \"5fd69affb622984ce7be4f827e482849e9b08a77f8753c60eaadd02360ec14b1\": rpc error: code = NotFound desc = could not find container \"5fd69affb622984ce7be4f827e482849e9b08a77f8753c60eaadd02360ec14b1\": container with ID starting with 5fd69affb622984ce7be4f827e482849e9b08a77f8753c60eaadd02360ec14b1 not found: ID does not exist"
Apr 16 17:17:42.053799 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:42.053775    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw"]
Apr 16 17:17:42.058542 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:42.058523    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-lh8dw"]
Apr 16 17:17:43.037533 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:43.037505    2578 generic.go:358] "Generic (PLEG): container finished" podID="5e5a51cf-608e-4c37-9b18-9fbfdd71494b" containerID="26577f15c89889fcc685efaadac7595b034da4e776f3b20aac3d7449c17390dd" exitCode=0
Apr 16 17:17:43.037885 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:43.037578    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5" event={"ID":"5e5a51cf-608e-4c37-9b18-9fbfdd71494b","Type":"ContainerDied","Data":"26577f15c89889fcc685efaadac7595b034da4e776f3b20aac3d7449c17390dd"}
Apr 16 17:17:43.235010 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:43.234982    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f6ecbab-a599-47f8-adbc-4ed01883b9a4" path="/var/lib/kubelet/pods/9f6ecbab-a599-47f8-adbc-4ed01883b9a4/volumes"
Apr 16 17:17:44.043189 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:44.043159    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5" event={"ID":"5e5a51cf-608e-4c37-9b18-9fbfdd71494b","Type":"ContainerStarted","Data":"c9223a846b6f3e557e834e306ec3243dc7498c77eaf5098ab14096329b7aedca"}
Apr 16 17:17:44.043545 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:44.043370    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5"
Apr 16 17:17:44.057879 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:17:44.057832    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5" podStartSLOduration=6.057819994 podStartE2EDuration="6.057819994s" podCreationTimestamp="2026-04-16 17:17:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:17:44.056832314 +0000 UTC m=+2873.454048702" watchObservedRunningTime="2026-04-16 17:17:44.057819994 +0000 UTC m=+2873.455036382"
Apr 16 17:18:15.130346 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:15.130306    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5"
Apr 16 17:18:18.518948 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:18.518916    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5"]
Apr 16 17:18:18.519304 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:18.519213    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5" podUID="5e5a51cf-608e-4c37-9b18-9fbfdd71494b" containerName="kserve-container" containerID="cri-o://c9223a846b6f3e557e834e306ec3243dc7498c77eaf5098ab14096329b7aedca" gracePeriod=30
Apr 16 17:18:18.588577 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:18.588535    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr"]
Apr 16 17:18:18.588922 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:18.588905    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f6ecbab-a599-47f8-adbc-4ed01883b9a4" containerName="storage-initializer"
Apr 16 17:18:18.589017 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:18.588925    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f6ecbab-a599-47f8-adbc-4ed01883b9a4" containerName="storage-initializer"
Apr 16 17:18:18.589017 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:18.588940    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f6ecbab-a599-47f8-adbc-4ed01883b9a4" containerName="kserve-container"
Apr 16 17:18:18.589017 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:18.588958    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f6ecbab-a599-47f8-adbc-4ed01883b9a4" containerName="kserve-container"
Apr 16 17:18:18.589181 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:18.589047    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="9f6ecbab-a599-47f8-adbc-4ed01883b9a4" containerName="kserve-container"
Apr 16 17:18:18.591891 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:18.591871    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr"
Apr 16 17:18:18.603935 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:18.603915    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr"]
Apr 16 17:18:18.735781 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:18.735757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/905c2bb3-7ccd-4202-84a5-50af814d2dcc-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-c58d48f-wwdlr\" (UID: \"905c2bb3-7ccd-4202-84a5-50af814d2dcc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr"
Apr 16 17:18:18.836797 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:18.836718    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/905c2bb3-7ccd-4202-84a5-50af814d2dcc-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-c58d48f-wwdlr\" (UID: \"905c2bb3-7ccd-4202-84a5-50af814d2dcc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr"
Apr 16 17:18:18.837079 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:18.837062    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/905c2bb3-7ccd-4202-84a5-50af814d2dcc-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-c58d48f-wwdlr\" (UID: \"905c2bb3-7ccd-4202-84a5-50af814d2dcc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr"
Apr 16 17:18:18.901311 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:18.901286    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr"
Apr 16 17:18:19.018295 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:19.018271    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr"]
Apr 16 17:18:19.020788 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:18:19.020760    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905c2bb3_7ccd_4202_84a5_50af814d2dcc.slice/crio-416c16db4ffebb76d68b1f1e03f2adab8d9643673691179fae2621469c37f013 WatchSource:0}: Error finding container 416c16db4ffebb76d68b1f1e03f2adab8d9643673691179fae2621469c37f013: Status 404 returned error can't find the container with id 416c16db4ffebb76d68b1f1e03f2adab8d9643673691179fae2621469c37f013
Apr 16 17:18:19.140688 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:19.140609    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr" event={"ID":"905c2bb3-7ccd-4202-84a5-50af814d2dcc","Type":"ContainerStarted","Data":"2a782ceef5abde5bd5e54a34c58204058aef3460c8ae74094ed1412aa32c6099"}
Apr 16 17:18:19.140688 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:19.140649    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr" event={"ID":"905c2bb3-7ccd-4202-84a5-50af814d2dcc","Type":"ContainerStarted","Data":"416c16db4ffebb76d68b1f1e03f2adab8d9643673691179fae2621469c37f013"}
Apr 16 17:18:23.152066 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:23.152033    2578 generic.go:358] "Generic (PLEG): container finished" podID="905c2bb3-7ccd-4202-84a5-50af814d2dcc" containerID="2a782ceef5abde5bd5e54a34c58204058aef3460c8ae74094ed1412aa32c6099" exitCode=0
Apr 16 17:18:23.152435 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:23.152107    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr" event={"ID":"905c2bb3-7ccd-4202-84a5-50af814d2dcc","Type":"ContainerDied","Data":"2a782ceef5abde5bd5e54a34c58204058aef3460c8ae74094ed1412aa32c6099"}
Apr 16 17:18:24.047591 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.047549    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5"
Apr 16 17:18:24.156295 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.156233    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr" event={"ID":"905c2bb3-7ccd-4202-84a5-50af814d2dcc","Type":"ContainerStarted","Data":"8a409d270e6799608d1b747824ce269d602c2d250b700312b8bb7fcd8d3f39b0"}
Apr 16 17:18:24.156657 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.156467    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr"
Apr 16 17:18:24.157547 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.157524    2578 generic.go:358] "Generic (PLEG): container finished" podID="5e5a51cf-608e-4c37-9b18-9fbfdd71494b" containerID="c9223a846b6f3e557e834e306ec3243dc7498c77eaf5098ab14096329b7aedca" exitCode=0
Apr 16 17:18:24.157646 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.157587    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5" event={"ID":"5e5a51cf-608e-4c37-9b18-9fbfdd71494b","Type":"ContainerDied","Data":"c9223a846b6f3e557e834e306ec3243dc7498c77eaf5098ab14096329b7aedca"}
Apr 16 17:18:24.157646 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.157618    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5" event={"ID":"5e5a51cf-608e-4c37-9b18-9fbfdd71494b","Type":"ContainerDied","Data":"dd2043f74f78068a1157514b9ba5d3d8f7afa84389be0cbccfc757bc9cc25739"}
Apr 16 17:18:24.157646 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.157637    2578 scope.go:117] "RemoveContainer" containerID="c9223a846b6f3e557e834e306ec3243dc7498c77eaf5098ab14096329b7aedca"
Apr 16 17:18:24.157789 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.157597    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5"
Apr 16 17:18:24.164855 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.164837    2578 scope.go:117] "RemoveContainer" containerID="26577f15c89889fcc685efaadac7595b034da4e776f3b20aac3d7449c17390dd"
Apr 16 17:18:24.171287 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.171268    2578 scope.go:117] "RemoveContainer" containerID="c9223a846b6f3e557e834e306ec3243dc7498c77eaf5098ab14096329b7aedca"
Apr 16 17:18:24.171671 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:18:24.171550    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9223a846b6f3e557e834e306ec3243dc7498c77eaf5098ab14096329b7aedca\": container with ID starting with c9223a846b6f3e557e834e306ec3243dc7498c77eaf5098ab14096329b7aedca not found: ID does not exist" containerID="c9223a846b6f3e557e834e306ec3243dc7498c77eaf5098ab14096329b7aedca"
Apr 16 17:18:24.171671 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.171598    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9223a846b6f3e557e834e306ec3243dc7498c77eaf5098ab14096329b7aedca"} err="failed to get container status \"c9223a846b6f3e557e834e306ec3243dc7498c77eaf5098ab14096329b7aedca\": rpc error: code = NotFound desc = could not find container \"c9223a846b6f3e557e834e306ec3243dc7498c77eaf5098ab14096329b7aedca\": container with ID starting with c9223a846b6f3e557e834e306ec3243dc7498c77eaf5098ab14096329b7aedca not found: ID does not exist"
Apr 16 17:18:24.171671 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.171621    2578 scope.go:117] "RemoveContainer" containerID="26577f15c89889fcc685efaadac7595b034da4e776f3b20aac3d7449c17390dd"
Apr 16 17:18:24.171895 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:18:24.171871    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26577f15c89889fcc685efaadac7595b034da4e776f3b20aac3d7449c17390dd\": container with ID starting with 26577f15c89889fcc685efaadac7595b034da4e776f3b20aac3d7449c17390dd not found: ID does not exist" containerID="26577f15c89889fcc685efaadac7595b034da4e776f3b20aac3d7449c17390dd"
Apr 16 17:18:24.171961 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.171904    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26577f15c89889fcc685efaadac7595b034da4e776f3b20aac3d7449c17390dd"} err="failed to get container status \"26577f15c89889fcc685efaadac7595b034da4e776f3b20aac3d7449c17390dd\": rpc error: code = NotFound desc = could not find container \"26577f15c89889fcc685efaadac7595b034da4e776f3b20aac3d7449c17390dd\": container with ID starting with 26577f15c89889fcc685efaadac7595b034da4e776f3b20aac3d7449c17390dd not found: ID does not exist"
Apr 16 17:18:24.173058 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.173023    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr" podStartSLOduration=6.173011961 podStartE2EDuration="6.173011961s" podCreationTimestamp="2026-04-16 17:18:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:18:24.171263955 +0000 UTC m=+2913.568480342" watchObservedRunningTime="2026-04-16 17:18:24.173011961 +0000 UTC m=+2913.570228348"
Apr 16 17:18:24.179252 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.179235    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e5a51cf-608e-4c37-9b18-9fbfdd71494b-kserve-provision-location\") pod \"5e5a51cf-608e-4c37-9b18-9fbfdd71494b\" (UID: \"5e5a51cf-608e-4c37-9b18-9fbfdd71494b\") "
Apr 16 17:18:24.179538 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.179517    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5a51cf-608e-4c37-9b18-9fbfdd71494b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e5a51cf-608e-4c37-9b18-9fbfdd71494b" (UID: "5e5a51cf-608e-4c37-9b18-9fbfdd71494b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:18:24.279754 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.279730    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e5a51cf-608e-4c37-9b18-9fbfdd71494b-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:18:24.478191 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.478165    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5"]
Apr 16 17:18:24.483288 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:24.483266    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-sf8b5"]
Apr 16 17:18:25.234673 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:25.234640    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e5a51cf-608e-4c37-9b18-9fbfdd71494b" path="/var/lib/kubelet/pods/5e5a51cf-608e-4c37-9b18-9fbfdd71494b/volumes"
Apr 16 17:18:55.229698 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:55.229667    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr"
Apr 16 17:18:58.684646 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:58.684613    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr"]
Apr 16 17:18:58.685017 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:58.684844    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr" podUID="905c2bb3-7ccd-4202-84a5-50af814d2dcc" containerName="kserve-container" containerID="cri-o://8a409d270e6799608d1b747824ce269d602c2d250b700312b8bb7fcd8d3f39b0" gracePeriod=30
Apr 16 17:18:58.728897 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:58.728869    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9"]
Apr 16 17:18:58.729211 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:58.729198    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e5a51cf-608e-4c37-9b18-9fbfdd71494b" containerName="kserve-container"
Apr 16 17:18:58.729257 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:58.729213    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e5a51cf-608e-4c37-9b18-9fbfdd71494b" containerName="kserve-container"
Apr 16 17:18:58.729257 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:58.729237    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e5a51cf-608e-4c37-9b18-9fbfdd71494b" containerName="storage-initializer"
Apr 16 17:18:58.729257 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:58.729242    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e5a51cf-608e-4c37-9b18-9fbfdd71494b" containerName="storage-initializer"
Apr 16 17:18:58.729358 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:58.729293    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="5e5a51cf-608e-4c37-9b18-9fbfdd71494b" containerName="kserve-container"
Apr 16 17:18:58.732237 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:58.732215    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9"
Apr 16 17:18:58.741997 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:58.741970    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9"]
Apr 16 17:18:58.812508 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:58.812481    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e93816f-a486-4e3b-9c3c-3f77b65b5159-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-687c7765c9-kxfj9\" (UID: \"5e93816f-a486-4e3b-9c3c-3f77b65b5159\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9"
Apr 16 17:18:58.912929 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:58.912905    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e93816f-a486-4e3b-9c3c-3f77b65b5159-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-687c7765c9-kxfj9\" (UID: \"5e93816f-a486-4e3b-9c3c-3f77b65b5159\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9"
Apr 16 17:18:58.913244 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:58.913225    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e93816f-a486-4e3b-9c3c-3f77b65b5159-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-687c7765c9-kxfj9\" (UID: \"5e93816f-a486-4e3b-9c3c-3f77b65b5159\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9"
Apr 16 17:18:59.044738 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:59.044715    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9"
Apr 16 17:18:59.160647 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:59.160614    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9"]
Apr 16 17:18:59.163784 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:18:59.163758    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e93816f_a486_4e3b_9c3c_3f77b65b5159.slice/crio-7344e71cae6ec2689fc4558321ec6a4b3e8cbc4d6f631c058903f509070e856e WatchSource:0}: Error finding container 7344e71cae6ec2689fc4558321ec6a4b3e8cbc4d6f631c058903f509070e856e: Status 404 returned error can't find the container with id 7344e71cae6ec2689fc4558321ec6a4b3e8cbc4d6f631c058903f509070e856e
Apr 16 17:18:59.257860 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:59.257791    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9" event={"ID":"5e93816f-a486-4e3b-9c3c-3f77b65b5159","Type":"ContainerStarted","Data":"2041e55cbd51d6b08d93fa1e0d35b8040bf4aa6c8c38924d6a82568f308959dc"}
Apr 16 17:18:59.257860 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:18:59.257824    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9" event={"ID":"5e93816f-a486-4e3b-9c3c-3f77b65b5159","Type":"ContainerStarted","Data":"7344e71cae6ec2689fc4558321ec6a4b3e8cbc4d6f631c058903f509070e856e"}
Apr 16 17:19:03.270742 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:03.270711    2578 generic.go:358] "Generic (PLEG): container finished" podID="5e93816f-a486-4e3b-9c3c-3f77b65b5159" containerID="2041e55cbd51d6b08d93fa1e0d35b8040bf4aa6c8c38924d6a82568f308959dc" exitCode=0
Apr 16 17:19:03.270742 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:03.270744    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9" event={"ID":"5e93816f-a486-4e3b-9c3c-3f77b65b5159","Type":"ContainerDied","Data":"2041e55cbd51d6b08d93fa1e0d35b8040bf4aa6c8c38924d6a82568f308959dc"}
Apr 16 17:19:04.224603 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.224583    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr"
Apr 16 17:19:04.257259 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.257237    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/905c2bb3-7ccd-4202-84a5-50af814d2dcc-kserve-provision-location\") pod \"905c2bb3-7ccd-4202-84a5-50af814d2dcc\" (UID: \"905c2bb3-7ccd-4202-84a5-50af814d2dcc\") "
Apr 16 17:19:04.257572 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.257529    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905c2bb3-7ccd-4202-84a5-50af814d2dcc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "905c2bb3-7ccd-4202-84a5-50af814d2dcc" (UID: "905c2bb3-7ccd-4202-84a5-50af814d2dcc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:19:04.275193 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.275163    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9" event={"ID":"5e93816f-a486-4e3b-9c3c-3f77b65b5159","Type":"ContainerStarted","Data":"8346dacdded3a8eba73b559f922d12fa14698f9ec5171da148a5b5ebf6fb09b6"}
Apr 16 17:19:04.275537 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.275457    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9"
Apr 16 17:19:04.276702 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.276674    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9" podUID="5e93816f-a486-4e3b-9c3c-3f77b65b5159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Apr 16 17:19:04.276820 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.276700    2578 generic.go:358] "Generic (PLEG): container finished" podID="905c2bb3-7ccd-4202-84a5-50af814d2dcc" containerID="8a409d270e6799608d1b747824ce269d602c2d250b700312b8bb7fcd8d3f39b0" exitCode=0
Apr 16 17:19:04.276820 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.276752    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr"
Apr 16 17:19:04.276820 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.276776    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr" event={"ID":"905c2bb3-7ccd-4202-84a5-50af814d2dcc","Type":"ContainerDied","Data":"8a409d270e6799608d1b747824ce269d602c2d250b700312b8bb7fcd8d3f39b0"}
Apr 16 17:19:04.276983 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.276817    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr" event={"ID":"905c2bb3-7ccd-4202-84a5-50af814d2dcc","Type":"ContainerDied","Data":"416c16db4ffebb76d68b1f1e03f2adab8d9643673691179fae2621469c37f013"}
Apr 16 17:19:04.276983 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.276843    2578 scope.go:117] "RemoveContainer" containerID="8a409d270e6799608d1b747824ce269d602c2d250b700312b8bb7fcd8d3f39b0"
Apr 16 17:19:04.284412 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.284380    2578 scope.go:117] "RemoveContainer" containerID="2a782ceef5abde5bd5e54a34c58204058aef3460c8ae74094ed1412aa32c6099"
Apr 16 17:19:04.291358 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.291323    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9" podStartSLOduration=6.291312072 podStartE2EDuration="6.291312072s" podCreationTimestamp="2026-04-16 17:18:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:19:04.289761016 +0000 UTC m=+2953.686977404" watchObservedRunningTime="2026-04-16 17:19:04.291312072 +0000 UTC m=+2953.688528460"
Apr 16 17:19:04.291619 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.291598    2578 scope.go:117] "RemoveContainer" containerID="8a409d270e6799608d1b747824ce269d602c2d250b700312b8bb7fcd8d3f39b0"
Apr 16 17:19:04.291846 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:19:04.291828    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a409d270e6799608d1b747824ce269d602c2d250b700312b8bb7fcd8d3f39b0\": container with ID starting with 8a409d270e6799608d1b747824ce269d602c2d250b700312b8bb7fcd8d3f39b0 not found: ID does not exist" containerID="8a409d270e6799608d1b747824ce269d602c2d250b700312b8bb7fcd8d3f39b0"
Apr 16 17:19:04.291911 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.291852    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a409d270e6799608d1b747824ce269d602c2d250b700312b8bb7fcd8d3f39b0"} err="failed to get container status \"8a409d270e6799608d1b747824ce269d602c2d250b700312b8bb7fcd8d3f39b0\": rpc error: code = NotFound desc = could not find container \"8a409d270e6799608d1b747824ce269d602c2d250b700312b8bb7fcd8d3f39b0\": container with ID starting with 8a409d270e6799608d1b747824ce269d602c2d250b700312b8bb7fcd8d3f39b0 not found: ID does not exist"
Apr 16 17:19:04.291911 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.291869    2578 scope.go:117] "RemoveContainer" containerID="2a782ceef5abde5bd5e54a34c58204058aef3460c8ae74094ed1412aa32c6099"
Apr 16 17:19:04.292099 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:19:04.292080    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a782ceef5abde5bd5e54a34c58204058aef3460c8ae74094ed1412aa32c6099\": container with ID starting with 2a782ceef5abde5bd5e54a34c58204058aef3460c8ae74094ed1412aa32c6099 not found: ID does not exist" containerID="2a782ceef5abde5bd5e54a34c58204058aef3460c8ae74094ed1412aa32c6099"
Apr 16 17:19:04.292156 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.292108    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a782ceef5abde5bd5e54a34c58204058aef3460c8ae74094ed1412aa32c6099"} err="failed to get container status \"2a782ceef5abde5bd5e54a34c58204058aef3460c8ae74094ed1412aa32c6099\": rpc error: code = NotFound desc = could not find container \"2a782ceef5abde5bd5e54a34c58204058aef3460c8ae74094ed1412aa32c6099\": container with ID starting with 2a782ceef5abde5bd5e54a34c58204058aef3460c8ae74094ed1412aa32c6099 not found: ID does not exist"
Apr 16 17:19:04.306891 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.306845    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr"]
Apr 16 17:19:04.309158 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.309136    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wwdlr"]
Apr 16 17:19:04.357936 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:04.357914    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/905c2bb3-7ccd-4202-84a5-50af814d2dcc-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:19:05.235515 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:05.235486    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="905c2bb3-7ccd-4202-84a5-50af814d2dcc" path="/var/lib/kubelet/pods/905c2bb3-7ccd-4202-84a5-50af814d2dcc/volumes"
Apr 16 17:19:05.280532 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:05.280496    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9" podUID="5e93816f-a486-4e3b-9c3c-3f77b65b5159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Apr 16 17:19:15.280803 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:15.280766    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9" podUID="5e93816f-a486-4e3b-9c3c-3f77b65b5159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Apr 16 17:19:25.280604 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:25.280546    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9" podUID="5e93816f-a486-4e3b-9c3c-3f77b65b5159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Apr 16 17:19:35.281318 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:35.281281    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9" podUID="5e93816f-a486-4e3b-9c3c-3f77b65b5159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Apr 16 17:19:45.280726 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:45.280685    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9" podUID="5e93816f-a486-4e3b-9c3c-3f77b65b5159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Apr 16 17:19:55.281390 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:19:55.281346    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9" podUID="5e93816f-a486-4e3b-9c3c-3f77b65b5159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Apr 16 17:20:05.281248 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:05.281217    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9"
Apr 16 17:20:08.869363 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:08.869334    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9"]
Apr 16 17:20:08.869824 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:08.869609    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9" podUID="5e93816f-a486-4e3b-9c3c-3f77b65b5159" containerName="kserve-container" containerID="cri-o://8346dacdded3a8eba73b559f922d12fa14698f9ec5171da148a5b5ebf6fb09b6" gracePeriod=30
Apr 16 17:20:08.967827 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:08.967797    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p"]
Apr 16 17:20:08.968132 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:08.968119    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="905c2bb3-7ccd-4202-84a5-50af814d2dcc" containerName="storage-initializer"
Apr 16 17:20:08.968182 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:08.968134    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="905c2bb3-7ccd-4202-84a5-50af814d2dcc" containerName="storage-initializer"
Apr 16 17:20:08.968182 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:08.968158    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="905c2bb3-7ccd-4202-84a5-50af814d2dcc" containerName="kserve-container"
Apr 16 17:20:08.968182 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:08.968166    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="905c2bb3-7ccd-4202-84a5-50af814d2dcc" containerName="kserve-container"
Apr 16 17:20:08.968305 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:08.968227    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="905c2bb3-7ccd-4202-84a5-50af814d2dcc" containerName="kserve-container"
Apr 16 17:20:08.972675 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:08.972657    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p"
Apr 16 17:20:08.978906 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:08.978884    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p"]
Apr 16 17:20:09.019809 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:09.019785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec153cf2-5835-4702-89ab-9d1874d93f98-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p\" (UID: \"ec153cf2-5835-4702-89ab-9d1874d93f98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p"
Apr 16 17:20:09.121186 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:09.121127    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec153cf2-5835-4702-89ab-9d1874d93f98-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p\" (UID: \"ec153cf2-5835-4702-89ab-9d1874d93f98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p"
Apr 16 17:20:09.121435 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:09.121419    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec153cf2-5835-4702-89ab-9d1874d93f98-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p\" (UID: \"ec153cf2-5835-4702-89ab-9d1874d93f98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p"
Apr 16 17:20:09.283593 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:09.283570    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p"
Apr 16 17:20:09.437329 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:09.437257    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p"]
Apr 16 17:20:09.440321 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:20:09.440295    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec153cf2_5835_4702_89ab_9d1874d93f98.slice/crio-f12b96db594664536bc4a2650e9d357cbc96ec8afa429f157b09e53d36870676 WatchSource:0}: Error finding container f12b96db594664536bc4a2650e9d357cbc96ec8afa429f157b09e53d36870676: Status 404 returned error can't find the container with id f12b96db594664536bc4a2650e9d357cbc96ec8afa429f157b09e53d36870676
Apr 16 17:20:09.463221 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:09.463180    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p" event={"ID":"ec153cf2-5835-4702-89ab-9d1874d93f98","Type":"ContainerStarted","Data":"f12b96db594664536bc4a2650e9d357cbc96ec8afa429f157b09e53d36870676"}
Apr 16 17:20:10.467050 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:10.467012    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p" event={"ID":"ec153cf2-5835-4702-89ab-9d1874d93f98","Type":"ContainerStarted","Data":"ad49cadb9c10bbafcf5e82ade7a9b4224710f0fb7821987ae43434d8f59d248e"}
Apr 16 17:20:12.003537 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:12.003516    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9"
Apr 16 17:20:12.044261 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:12.044238    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e93816f-a486-4e3b-9c3c-3f77b65b5159-kserve-provision-location\") pod \"5e93816f-a486-4e3b-9c3c-3f77b65b5159\" (UID: \"5e93816f-a486-4e3b-9c3c-3f77b65b5159\") "
Apr 16 17:20:12.044548 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:12.044525    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e93816f-a486-4e3b-9c3c-3f77b65b5159-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e93816f-a486-4e3b-9c3c-3f77b65b5159" (UID: "5e93816f-a486-4e3b-9c3c-3f77b65b5159"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:20:12.145178 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:12.145111    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e93816f-a486-4e3b-9c3c-3f77b65b5159-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:20:12.473259 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:12.473224    2578 generic.go:358] "Generic (PLEG): container finished" podID="5e93816f-a486-4e3b-9c3c-3f77b65b5159" containerID="8346dacdded3a8eba73b559f922d12fa14698f9ec5171da148a5b5ebf6fb09b6" exitCode=0
Apr 16 17:20:12.473420 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:12.473298    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9" event={"ID":"5e93816f-a486-4e3b-9c3c-3f77b65b5159","Type":"ContainerDied","Data":"8346dacdded3a8eba73b559f922d12fa14698f9ec5171da148a5b5ebf6fb09b6"}
Apr 16 17:20:12.473420 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:12.473308    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9"
Apr 16 17:20:12.473420 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:12.473324    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9" event={"ID":"5e93816f-a486-4e3b-9c3c-3f77b65b5159","Type":"ContainerDied","Data":"7344e71cae6ec2689fc4558321ec6a4b3e8cbc4d6f631c058903f509070e856e"}
Apr 16 17:20:12.473420 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:12.473338    2578 scope.go:117] "RemoveContainer" containerID="8346dacdded3a8eba73b559f922d12fa14698f9ec5171da148a5b5ebf6fb09b6"
Apr 16 17:20:12.482005 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:12.481989    2578 scope.go:117] "RemoveContainer" containerID="2041e55cbd51d6b08d93fa1e0d35b8040bf4aa6c8c38924d6a82568f308959dc"
Apr 16 17:20:12.488753 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:12.488739    2578 scope.go:117] "RemoveContainer" containerID="8346dacdded3a8eba73b559f922d12fa14698f9ec5171da148a5b5ebf6fb09b6"
Apr 16 17:20:12.488993 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:20:12.488965    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8346dacdded3a8eba73b559f922d12fa14698f9ec5171da148a5b5ebf6fb09b6\": container with ID starting with 8346dacdded3a8eba73b559f922d12fa14698f9ec5171da148a5b5ebf6fb09b6 not found: ID does not exist" containerID="8346dacdded3a8eba73b559f922d12fa14698f9ec5171da148a5b5ebf6fb09b6"
Apr 16 17:20:12.489044 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:12.489000    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8346dacdded3a8eba73b559f922d12fa14698f9ec5171da148a5b5ebf6fb09b6"} err="failed to get container status \"8346dacdded3a8eba73b559f922d12fa14698f9ec5171da148a5b5ebf6fb09b6\": rpc error: code = NotFound desc = could not find container \"8346dacdded3a8eba73b559f922d12fa14698f9ec5171da148a5b5ebf6fb09b6\": container with ID starting with 8346dacdded3a8eba73b559f922d12fa14698f9ec5171da148a5b5ebf6fb09b6 not found: ID does not exist"
Apr 16 17:20:12.489044 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:12.489022    2578 scope.go:117] "RemoveContainer" containerID="2041e55cbd51d6b08d93fa1e0d35b8040bf4aa6c8c38924d6a82568f308959dc"
Apr 16 17:20:12.489239 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:20:12.489217    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2041e55cbd51d6b08d93fa1e0d35b8040bf4aa6c8c38924d6a82568f308959dc\": container with ID starting with 2041e55cbd51d6b08d93fa1e0d35b8040bf4aa6c8c38924d6a82568f308959dc not found: ID does not exist" containerID="2041e55cbd51d6b08d93fa1e0d35b8040bf4aa6c8c38924d6a82568f308959dc"
Apr 16 17:20:12.489301 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:12.489249    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2041e55cbd51d6b08d93fa1e0d35b8040bf4aa6c8c38924d6a82568f308959dc"} err="failed to get container status \"2041e55cbd51d6b08d93fa1e0d35b8040bf4aa6c8c38924d6a82568f308959dc\": rpc error: code = NotFound desc = could not find container \"2041e55cbd51d6b08d93fa1e0d35b8040bf4aa6c8c38924d6a82568f308959dc\": container with ID starting with 2041e55cbd51d6b08d93fa1e0d35b8040bf4aa6c8c38924d6a82568f308959dc not found: ID does not exist"
Apr 16 17:20:12.493524 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:12.493501    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9"]
Apr 16 17:20:12.496988 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:12.496961    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-kxfj9"]
Apr 16 17:20:13.235755 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:13.235724    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e93816f-a486-4e3b-9c3c-3f77b65b5159" path="/var/lib/kubelet/pods/5e93816f-a486-4e3b-9c3c-3f77b65b5159/volumes"
Apr 16 17:20:13.282402 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:20:13.282376    2578 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec153cf2_5835_4702_89ab_9d1874d93f98.slice/crio-conmon-ad49cadb9c10bbafcf5e82ade7a9b4224710f0fb7821987ae43434d8f59d248e.scope\": RecentStats: unable to find data in memory cache]"
Apr 16 17:20:13.477076 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:13.477050    2578 generic.go:358] "Generic (PLEG): container finished" podID="ec153cf2-5835-4702-89ab-9d1874d93f98" containerID="ad49cadb9c10bbafcf5e82ade7a9b4224710f0fb7821987ae43434d8f59d248e" exitCode=0
Apr 16 17:20:13.477176 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:13.477121    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p" event={"ID":"ec153cf2-5835-4702-89ab-9d1874d93f98","Type":"ContainerDied","Data":"ad49cadb9c10bbafcf5e82ade7a9b4224710f0fb7821987ae43434d8f59d248e"}
Apr 16 17:20:14.482892 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:14.482860    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p" event={"ID":"ec153cf2-5835-4702-89ab-9d1874d93f98","Type":"ContainerStarted","Data":"97ce82b61fe1ee57ceefcfffd6b21d6dcfa3d9151f9033eec11839908beae95c"}
Apr 16 17:20:14.483273 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:14.483069    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p"
Apr 16 17:20:14.500107 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:14.500058    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p" podStartSLOduration=6.50004736 podStartE2EDuration="6.50004736s" podCreationTimestamp="2026-04-16 17:20:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:20:14.499290499 +0000 UTC m=+3023.896506916" watchObservedRunningTime="2026-04-16 17:20:14.50004736 +0000 UTC m=+3023.897263748"
Apr 16 17:20:44.721295 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:44.721217    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 17:20:44.722187 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:44.722164    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 17:20:45.531393 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:45.531348    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p" podUID="ec153cf2-5835-4702-89ab-9d1874d93f98" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Apr 16 17:20:55.487495 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:55.487464    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p"
Apr 16 17:20:59.007999 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.007968    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p"]
Apr 16 17:20:59.008356 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.008284    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p" podUID="ec153cf2-5835-4702-89ab-9d1874d93f98" containerName="kserve-container" containerID="cri-o://97ce82b61fe1ee57ceefcfffd6b21d6dcfa3d9151f9033eec11839908beae95c" gracePeriod=30
Apr 16 17:20:59.058726 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.058692    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9"]
Apr 16 17:20:59.058992 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.058981    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e93816f-a486-4e3b-9c3c-3f77b65b5159" containerName="storage-initializer"
Apr 16 17:20:59.059037 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.058994    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e93816f-a486-4e3b-9c3c-3f77b65b5159" containerName="storage-initializer"
Apr 16 17:20:59.059037 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.059020    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e93816f-a486-4e3b-9c3c-3f77b65b5159" containerName="kserve-container"
Apr 16 17:20:59.059037 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.059026    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e93816f-a486-4e3b-9c3c-3f77b65b5159" containerName="kserve-container"
Apr 16 17:20:59.059130 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.059078    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="5e93816f-a486-4e3b-9c3c-3f77b65b5159" containerName="kserve-container"
Apr 16 17:20:59.061953 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.061937    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9"
Apr 16 17:20:59.070734 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.070709    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9"]
Apr 16 17:20:59.173984 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.173950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0757f819-8f70-402c-8b64-83c118d5d066-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-5db5686f9f-rhsn9\" (UID: \"0757f819-8f70-402c-8b64-83c118d5d066\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9"
Apr 16 17:20:59.274415 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.274343    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0757f819-8f70-402c-8b64-83c118d5d066-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-5db5686f9f-rhsn9\" (UID: \"0757f819-8f70-402c-8b64-83c118d5d066\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9"
Apr 16 17:20:59.274702 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.274685    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0757f819-8f70-402c-8b64-83c118d5d066-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-5db5686f9f-rhsn9\" (UID: \"0757f819-8f70-402c-8b64-83c118d5d066\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9"
Apr 16 17:20:59.372485 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.372438    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9"
Apr 16 17:20:59.489070 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.489048    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9"]
Apr 16 17:20:59.491649 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:20:59.491617    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0757f819_8f70_402c_8b64_83c118d5d066.slice/crio-5f64226ec75a0992ec73c28fe3ac1425ba90ff27a8f43181cc045de119e04ea0 WatchSource:0}: Error finding container 5f64226ec75a0992ec73c28fe3ac1425ba90ff27a8f43181cc045de119e04ea0: Status 404 returned error can't find the container with id 5f64226ec75a0992ec73c28fe3ac1425ba90ff27a8f43181cc045de119e04ea0
Apr 16 17:20:59.619379 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.619343    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9" event={"ID":"0757f819-8f70-402c-8b64-83c118d5d066","Type":"ContainerStarted","Data":"991b6430b9331dc8b735092b0dc6375ce7b977eb609aa36573e61b04ff228c48"}
Apr 16 17:20:59.619502 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:20:59.619391    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9" event={"ID":"0757f819-8f70-402c-8b64-83c118d5d066","Type":"ContainerStarted","Data":"5f64226ec75a0992ec73c28fe3ac1425ba90ff27a8f43181cc045de119e04ea0"}
Apr 16 17:21:03.631547 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:03.631513    2578 generic.go:358] "Generic (PLEG): container finished" podID="0757f819-8f70-402c-8b64-83c118d5d066" containerID="991b6430b9331dc8b735092b0dc6375ce7b977eb609aa36573e61b04ff228c48" exitCode=0
Apr 16 17:21:03.631944 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:03.631601    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9" event={"ID":"0757f819-8f70-402c-8b64-83c118d5d066","Type":"ContainerDied","Data":"991b6430b9331dc8b735092b0dc6375ce7b977eb609aa36573e61b04ff228c48"}
Apr 16 17:21:04.636466 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:04.636430    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9" event={"ID":"0757f819-8f70-402c-8b64-83c118d5d066","Type":"ContainerStarted","Data":"8257992a8e49e7bf16ab78037eb07ebf1e7568f46f6a38a64958ccb162166d9b"}
Apr 16 17:21:04.636905 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:04.636722    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9"
Apr 16 17:21:04.638342 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:04.638309    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9" podUID="0757f819-8f70-402c-8b64-83c118d5d066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Apr 16 17:21:04.651323 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:04.651284    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9" podStartSLOduration=5.65126871 podStartE2EDuration="5.65126871s" podCreationTimestamp="2026-04-16 17:20:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:21:04.650885913 +0000 UTC m=+3074.048102303" watchObservedRunningTime="2026-04-16 17:21:04.65126871 +0000 UTC m=+3074.048485098"
Apr 16 17:21:05.493218 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.493196    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p"
Apr 16 17:21:05.628822 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.628791    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec153cf2-5835-4702-89ab-9d1874d93f98-kserve-provision-location\") pod \"ec153cf2-5835-4702-89ab-9d1874d93f98\" (UID: \"ec153cf2-5835-4702-89ab-9d1874d93f98\") "
Apr 16 17:21:05.629090 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.629068    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec153cf2-5835-4702-89ab-9d1874d93f98-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec153cf2-5835-4702-89ab-9d1874d93f98" (UID: "ec153cf2-5835-4702-89ab-9d1874d93f98"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:21:05.640775 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.640746    2578 generic.go:358] "Generic (PLEG): container finished" podID="ec153cf2-5835-4702-89ab-9d1874d93f98" containerID="97ce82b61fe1ee57ceefcfffd6b21d6dcfa3d9151f9033eec11839908beae95c" exitCode=0
Apr 16 17:21:05.641137 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.640807    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p"
Apr 16 17:21:05.641137 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.640828    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p" event={"ID":"ec153cf2-5835-4702-89ab-9d1874d93f98","Type":"ContainerDied","Data":"97ce82b61fe1ee57ceefcfffd6b21d6dcfa3d9151f9033eec11839908beae95c"}
Apr 16 17:21:05.641137 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.640864    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p" event={"ID":"ec153cf2-5835-4702-89ab-9d1874d93f98","Type":"ContainerDied","Data":"f12b96db594664536bc4a2650e9d357cbc96ec8afa429f157b09e53d36870676"}
Apr 16 17:21:05.641137 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.640877    2578 scope.go:117] "RemoveContainer" containerID="97ce82b61fe1ee57ceefcfffd6b21d6dcfa3d9151f9033eec11839908beae95c"
Apr 16 17:21:05.641327 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.641196    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9" podUID="0757f819-8f70-402c-8b64-83c118d5d066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Apr 16 17:21:05.649057 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.649041    2578 scope.go:117] "RemoveContainer" containerID="ad49cadb9c10bbafcf5e82ade7a9b4224710f0fb7821987ae43434d8f59d248e"
Apr 16 17:21:05.655861 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.655846    2578 scope.go:117] "RemoveContainer" containerID="97ce82b61fe1ee57ceefcfffd6b21d6dcfa3d9151f9033eec11839908beae95c"
Apr 16 17:21:05.656091 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:21:05.656076    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97ce82b61fe1ee57ceefcfffd6b21d6dcfa3d9151f9033eec11839908beae95c\": container with ID starting with 97ce82b61fe1ee57ceefcfffd6b21d6dcfa3d9151f9033eec11839908beae95c not found: ID does not exist" containerID="97ce82b61fe1ee57ceefcfffd6b21d6dcfa3d9151f9033eec11839908beae95c"
Apr 16 17:21:05.656135 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.656097    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97ce82b61fe1ee57ceefcfffd6b21d6dcfa3d9151f9033eec11839908beae95c"} err="failed to get container status \"97ce82b61fe1ee57ceefcfffd6b21d6dcfa3d9151f9033eec11839908beae95c\": rpc error: code = NotFound desc = could not find container \"97ce82b61fe1ee57ceefcfffd6b21d6dcfa3d9151f9033eec11839908beae95c\": container with ID starting with 97ce82b61fe1ee57ceefcfffd6b21d6dcfa3d9151f9033eec11839908beae95c not found: ID does not exist"
Apr 16 17:21:05.656135 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.656112    2578 scope.go:117] "RemoveContainer" containerID="ad49cadb9c10bbafcf5e82ade7a9b4224710f0fb7821987ae43434d8f59d248e"
Apr 16 17:21:05.656347 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:21:05.656329    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad49cadb9c10bbafcf5e82ade7a9b4224710f0fb7821987ae43434d8f59d248e\": container with ID starting with ad49cadb9c10bbafcf5e82ade7a9b4224710f0fb7821987ae43434d8f59d248e not found: ID does not exist" containerID="ad49cadb9c10bbafcf5e82ade7a9b4224710f0fb7821987ae43434d8f59d248e"
Apr 16 17:21:05.656385 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.656354    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad49cadb9c10bbafcf5e82ade7a9b4224710f0fb7821987ae43434d8f59d248e"} err="failed to get container status \"ad49cadb9c10bbafcf5e82ade7a9b4224710f0fb7821987ae43434d8f59d248e\": rpc error: code = NotFound desc = could not find container \"ad49cadb9c10bbafcf5e82ade7a9b4224710f0fb7821987ae43434d8f59d248e\": container with ID starting with ad49cadb9c10bbafcf5e82ade7a9b4224710f0fb7821987ae43434d8f59d248e not found: ID does not exist"
Apr 16 17:21:05.660820 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.660800    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p"]
Apr 16 17:21:05.663977 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.663958    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p"]
Apr 16 17:21:05.730035 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:05.730006    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec153cf2-5835-4702-89ab-9d1874d93f98-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:21:06.486635 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:06.486587    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-p2l6p" podUID="ec153cf2-5835-4702-89ab-9d1874d93f98" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.64:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Apr 16 17:21:07.236181 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:07.236143    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec153cf2-5835-4702-89ab-9d1874d93f98" path="/var/lib/kubelet/pods/ec153cf2-5835-4702-89ab-9d1874d93f98/volumes"
Apr 16 17:21:15.642078 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:15.642000    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9" podUID="0757f819-8f70-402c-8b64-83c118d5d066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Apr 16 17:21:25.641457 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:25.641413    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9" podUID="0757f819-8f70-402c-8b64-83c118d5d066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Apr 16 17:21:35.641109 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:35.641069    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9" podUID="0757f819-8f70-402c-8b64-83c118d5d066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Apr 16 17:21:45.641619 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:45.641577    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9" podUID="0757f819-8f70-402c-8b64-83c118d5d066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Apr 16 17:21:55.641214 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:21:55.641172    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9" podUID="0757f819-8f70-402c-8b64-83c118d5d066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Apr 16 17:22:05.642435 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:05.642405    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9"
Apr 16 17:22:09.163142 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.163111    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9"]
Apr 16 17:22:09.163548 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.163444    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9" podUID="0757f819-8f70-402c-8b64-83c118d5d066" containerName="kserve-container" containerID="cri-o://8257992a8e49e7bf16ab78037eb07ebf1e7568f46f6a38a64958ccb162166d9b" gracePeriod=30
Apr 16 17:22:09.244939 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.244883    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f"]
Apr 16 17:22:09.245200 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.245187    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec153cf2-5835-4702-89ab-9d1874d93f98" containerName="kserve-container"
Apr 16 17:22:09.245250 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.245201    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec153cf2-5835-4702-89ab-9d1874d93f98" containerName="kserve-container"
Apr 16 17:22:09.245250 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.245221    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec153cf2-5835-4702-89ab-9d1874d93f98" containerName="storage-initializer"
Apr 16 17:22:09.245250 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.245230    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec153cf2-5835-4702-89ab-9d1874d93f98" containerName="storage-initializer"
Apr 16 17:22:09.245352 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.245289    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="ec153cf2-5835-4702-89ab-9d1874d93f98" containerName="kserve-container"
Apr 16 17:22:09.248408 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.248393    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f"
Apr 16 17:22:09.251417 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.251396    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Apr 16 17:22:09.256855 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.256832    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f"]
Apr 16 17:22:09.274858 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.274837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3be2aca-401d-42fb-af06-e08e1ce0a152-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-695b7cc5c-pjr4f\" (UID: \"b3be2aca-401d-42fb-af06-e08e1ce0a152\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f"
Apr 16 17:22:09.375974 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.375952    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3be2aca-401d-42fb-af06-e08e1ce0a152-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-695b7cc5c-pjr4f\" (UID: \"b3be2aca-401d-42fb-af06-e08e1ce0a152\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f"
Apr 16 17:22:09.376266 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.376250    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3be2aca-401d-42fb-af06-e08e1ce0a152-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-695b7cc5c-pjr4f\" (UID: \"b3be2aca-401d-42fb-af06-e08e1ce0a152\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f"
Apr 16 17:22:09.559169 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.559145    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f"
Apr 16 17:22:09.673326 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.673296    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f"]
Apr 16 17:22:09.676914 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:22:09.676878    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3be2aca_401d_42fb_af06_e08e1ce0a152.slice/crio-3c2f9772aa0d8607aee2f87cf039901180293ba94980b9839f650ccd8d475bb5 WatchSource:0}: Error finding container 3c2f9772aa0d8607aee2f87cf039901180293ba94980b9839f650ccd8d475bb5: Status 404 returned error can't find the container with id 3c2f9772aa0d8607aee2f87cf039901180293ba94980b9839f650ccd8d475bb5
Apr 16 17:22:09.679051 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.679032    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 17:22:09.830349 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.830269    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f" event={"ID":"b3be2aca-401d-42fb-af06-e08e1ce0a152","Type":"ContainerStarted","Data":"327e31c95cae7ed1ec45a657c15bf265bcf4625f1e4c77cb3eb48c3e551d2cc4"}
Apr 16 17:22:09.830349 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:09.830309    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f" event={"ID":"b3be2aca-401d-42fb-af06-e08e1ce0a152","Type":"ContainerStarted","Data":"3c2f9772aa0d8607aee2f87cf039901180293ba94980b9839f650ccd8d475bb5"}
Apr 16 17:22:10.834940 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:10.834910    2578 generic.go:358] "Generic (PLEG): container finished" podID="b3be2aca-401d-42fb-af06-e08e1ce0a152" containerID="327e31c95cae7ed1ec45a657c15bf265bcf4625f1e4c77cb3eb48c3e551d2cc4" exitCode=0
Apr 16 17:22:10.835274 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:10.834983    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f" event={"ID":"b3be2aca-401d-42fb-af06-e08e1ce0a152","Type":"ContainerDied","Data":"327e31c95cae7ed1ec45a657c15bf265bcf4625f1e4c77cb3eb48c3e551d2cc4"}
Apr 16 17:22:11.839078 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:11.839043    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f" event={"ID":"b3be2aca-401d-42fb-af06-e08e1ce0a152","Type":"ContainerStarted","Data":"3469fd6778f1f99bf469a1370fee124a61a7580421f2441b87866ee2ad9aa1d5"}
Apr 16 17:22:11.839502 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:11.839266    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f"
Apr 16 17:22:11.840489 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:11.840467    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f" podUID="b3be2aca-401d-42fb-af06-e08e1ce0a152" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Apr 16 17:22:11.856981 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:11.856941    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f" podStartSLOduration=2.85692768 podStartE2EDuration="2.85692768s" podCreationTimestamp="2026-04-16 17:22:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:22:11.855883926 +0000 UTC m=+3141.253100337" watchObservedRunningTime="2026-04-16 17:22:11.85692768 +0000 UTC m=+3141.254144068"
Apr 16 17:22:12.601302 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.601276    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9"
Apr 16 17:22:12.700163 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.700144    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0757f819-8f70-402c-8b64-83c118d5d066-kserve-provision-location\") pod \"0757f819-8f70-402c-8b64-83c118d5d066\" (UID: \"0757f819-8f70-402c-8b64-83c118d5d066\") "
Apr 16 17:22:12.702001 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.700756    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0757f819-8f70-402c-8b64-83c118d5d066-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0757f819-8f70-402c-8b64-83c118d5d066" (UID: "0757f819-8f70-402c-8b64-83c118d5d066"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:22:12.801540 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.801514    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0757f819-8f70-402c-8b64-83c118d5d066-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:22:12.846048 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.846015    2578 generic.go:358] "Generic (PLEG): container finished" podID="0757f819-8f70-402c-8b64-83c118d5d066" containerID="8257992a8e49e7bf16ab78037eb07ebf1e7568f46f6a38a64958ccb162166d9b" exitCode=0
Apr 16 17:22:12.846432 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.846087    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9" event={"ID":"0757f819-8f70-402c-8b64-83c118d5d066","Type":"ContainerDied","Data":"8257992a8e49e7bf16ab78037eb07ebf1e7568f46f6a38a64958ccb162166d9b"}
Apr 16 17:22:12.846432 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.846116    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9"
Apr 16 17:22:12.846432 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.846135    2578 scope.go:117] "RemoveContainer" containerID="8257992a8e49e7bf16ab78037eb07ebf1e7568f46f6a38a64958ccb162166d9b"
Apr 16 17:22:12.846432 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.846122    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9" event={"ID":"0757f819-8f70-402c-8b64-83c118d5d066","Type":"ContainerDied","Data":"5f64226ec75a0992ec73c28fe3ac1425ba90ff27a8f43181cc045de119e04ea0"}
Apr 16 17:22:12.846776 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.846745    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f" podUID="b3be2aca-401d-42fb-af06-e08e1ce0a152" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Apr 16 17:22:12.854801 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.854776    2578 scope.go:117] "RemoveContainer" containerID="991b6430b9331dc8b735092b0dc6375ce7b977eb609aa36573e61b04ff228c48"
Apr 16 17:22:12.862005 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.861990    2578 scope.go:117] "RemoveContainer" containerID="8257992a8e49e7bf16ab78037eb07ebf1e7568f46f6a38a64958ccb162166d9b"
Apr 16 17:22:12.862262 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:22:12.862242    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8257992a8e49e7bf16ab78037eb07ebf1e7568f46f6a38a64958ccb162166d9b\": container with ID starting with 8257992a8e49e7bf16ab78037eb07ebf1e7568f46f6a38a64958ccb162166d9b not found: ID does not exist" containerID="8257992a8e49e7bf16ab78037eb07ebf1e7568f46f6a38a64958ccb162166d9b"
Apr 16 17:22:12.862336 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.862268    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8257992a8e49e7bf16ab78037eb07ebf1e7568f46f6a38a64958ccb162166d9b"} err="failed to get container status \"8257992a8e49e7bf16ab78037eb07ebf1e7568f46f6a38a64958ccb162166d9b\": rpc error: code = NotFound desc = could not find container \"8257992a8e49e7bf16ab78037eb07ebf1e7568f46f6a38a64958ccb162166d9b\": container with ID starting with 8257992a8e49e7bf16ab78037eb07ebf1e7568f46f6a38a64958ccb162166d9b not found: ID does not exist"
Apr 16 17:22:12.862336 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.862284    2578 scope.go:117] "RemoveContainer" containerID="991b6430b9331dc8b735092b0dc6375ce7b977eb609aa36573e61b04ff228c48"
Apr 16 17:22:12.862523 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:22:12.862507    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991b6430b9331dc8b735092b0dc6375ce7b977eb609aa36573e61b04ff228c48\": container with ID starting with 991b6430b9331dc8b735092b0dc6375ce7b977eb609aa36573e61b04ff228c48 not found: ID does not exist" containerID="991b6430b9331dc8b735092b0dc6375ce7b977eb609aa36573e61b04ff228c48"
Apr 16 17:22:12.862595 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.862528    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991b6430b9331dc8b735092b0dc6375ce7b977eb609aa36573e61b04ff228c48"} err="failed to get container status \"991b6430b9331dc8b735092b0dc6375ce7b977eb609aa36573e61b04ff228c48\": rpc error: code = NotFound desc = could not find container \"991b6430b9331dc8b735092b0dc6375ce7b977eb609aa36573e61b04ff228c48\": container with ID starting with 991b6430b9331dc8b735092b0dc6375ce7b977eb609aa36573e61b04ff228c48 not found: ID does not exist"
Apr 16 17:22:12.866475 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.866453    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9"]
Apr 16 17:22:12.870023 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:12.870004    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-rhsn9"]
Apr 16 17:22:13.235682 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:13.235656    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0757f819-8f70-402c-8b64-83c118d5d066" path="/var/lib/kubelet/pods/0757f819-8f70-402c-8b64-83c118d5d066/volumes"
Apr 16 17:22:22.847428 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:22.847383    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f" podUID="b3be2aca-401d-42fb-af06-e08e1ce0a152" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Apr 16 17:22:32.847635 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:32.847589    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f" podUID="b3be2aca-401d-42fb-af06-e08e1ce0a152" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Apr 16 17:22:42.847385 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:42.847344    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f" podUID="b3be2aca-401d-42fb-af06-e08e1ce0a152" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Apr 16 17:22:52.846880 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:22:52.846839    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f" podUID="b3be2aca-401d-42fb-af06-e08e1ce0a152" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Apr 16 17:23:02.846878 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:02.846838    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f" podUID="b3be2aca-401d-42fb-af06-e08e1ce0a152" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Apr 16 17:23:12.848200 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:12.848169    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f"
Apr 16 17:23:19.360145 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.360111    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f"]
Apr 16 17:23:19.360628 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.360442    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f" podUID="b3be2aca-401d-42fb-af06-e08e1ce0a152" containerName="kserve-container" containerID="cri-o://3469fd6778f1f99bf469a1370fee124a61a7580421f2441b87866ee2ad9aa1d5" gracePeriod=30
Apr 16 17:23:19.460377 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.460345    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"]
Apr 16 17:23:19.460681 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.460669    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0757f819-8f70-402c-8b64-83c118d5d066" containerName="kserve-container"
Apr 16 17:23:19.460744 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.460683    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0757f819-8f70-402c-8b64-83c118d5d066" containerName="kserve-container"
Apr 16 17:23:19.460744 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.460698    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0757f819-8f70-402c-8b64-83c118d5d066" containerName="storage-initializer"
Apr 16 17:23:19.460744 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.460704    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0757f819-8f70-402c-8b64-83c118d5d066" containerName="storage-initializer"
Apr 16 17:23:19.460851 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.460779    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="0757f819-8f70-402c-8b64-83c118d5d066" containerName="kserve-container"
Apr 16 17:23:19.464943 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.464928    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"
Apr 16 17:23:19.467220 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.467201    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Apr 16 17:23:19.469642 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.469619    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"]
Apr 16 17:23:19.568084 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.568049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43cfa70b-8fc2-4271-80c2-7ceb6e264fa9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb\" (UID: \"43cfa70b-8fc2-4271-80c2-7ceb6e264fa9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"
Apr 16 17:23:19.568212 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.568097    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43cfa70b-8fc2-4271-80c2-7ceb6e264fa9-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb\" (UID: \"43cfa70b-8fc2-4271-80c2-7ceb6e264fa9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"
Apr 16 17:23:19.669512 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.669434    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43cfa70b-8fc2-4271-80c2-7ceb6e264fa9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb\" (UID: \"43cfa70b-8fc2-4271-80c2-7ceb6e264fa9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"
Apr 16 17:23:19.669512 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.669475    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43cfa70b-8fc2-4271-80c2-7ceb6e264fa9-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb\" (UID: \"43cfa70b-8fc2-4271-80c2-7ceb6e264fa9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"
Apr 16 17:23:19.669861 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.669844    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43cfa70b-8fc2-4271-80c2-7ceb6e264fa9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb\" (UID: \"43cfa70b-8fc2-4271-80c2-7ceb6e264fa9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"
Apr 16 17:23:19.670018 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.670002    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43cfa70b-8fc2-4271-80c2-7ceb6e264fa9-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb\" (UID: \"43cfa70b-8fc2-4271-80c2-7ceb6e264fa9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"
Apr 16 17:23:19.775751 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.775721    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"
Apr 16 17:23:19.891257 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:19.891222    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"]
Apr 16 17:23:19.894378 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:23:19.894351    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cfa70b_8fc2_4271_80c2_7ceb6e264fa9.slice/crio-339d81b286449d885c21e201d485183af7565ad4eda26f015247e35665f5d517 WatchSource:0}: Error finding container 339d81b286449d885c21e201d485183af7565ad4eda26f015247e35665f5d517: Status 404 returned error can't find the container with id 339d81b286449d885c21e201d485183af7565ad4eda26f015247e35665f5d517
Apr 16 17:23:20.035215 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:20.035176    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb" event={"ID":"43cfa70b-8fc2-4271-80c2-7ceb6e264fa9","Type":"ContainerStarted","Data":"57812057cc2031a823a5a5aa8a5ac620874ba3bfc9440d179abe67d6872f3cbe"}
Apr 16 17:23:20.035215 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:20.035211    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb" event={"ID":"43cfa70b-8fc2-4271-80c2-7ceb6e264fa9","Type":"ContainerStarted","Data":"339d81b286449d885c21e201d485183af7565ad4eda26f015247e35665f5d517"}
Apr 16 17:23:21.039177 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:21.039149    2578 generic.go:358] "Generic (PLEG): container finished" podID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" containerID="57812057cc2031a823a5a5aa8a5ac620874ba3bfc9440d179abe67d6872f3cbe" exitCode=0
Apr 16 17:23:21.039542 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:21.039191    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb" event={"ID":"43cfa70b-8fc2-4271-80c2-7ceb6e264fa9","Type":"ContainerDied","Data":"57812057cc2031a823a5a5aa8a5ac620874ba3bfc9440d179abe67d6872f3cbe"}
Apr 16 17:23:22.042957 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:22.042921    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb" event={"ID":"43cfa70b-8fc2-4271-80c2-7ceb6e264fa9","Type":"ContainerStarted","Data":"7e4c46b0e2b3f4c716c4d22eca0f0aa561a0f4d3a26937a598e94cf02df3af57"}
Apr 16 17:23:22.043342 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:22.043157    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"
Apr 16 17:23:22.044394 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:22.044366    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb" podUID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Apr 16 17:23:22.060438 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:22.060400    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb" podStartSLOduration=3.060387448 podStartE2EDuration="3.060387448s" podCreationTimestamp="2026-04-16 17:23:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:23:22.058868823 +0000 UTC m=+3211.456085210" watchObservedRunningTime="2026-04-16 17:23:22.060387448 +0000 UTC m=+3211.457603838"
Apr 16 17:23:22.847715 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:22.847676    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f" podUID="b3be2aca-401d-42fb-af06-e08e1ce0a152" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Apr 16 17:23:23.046788 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:23.046753    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb" podUID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Apr 16 17:23:23.290793 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:23.290772    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f"
Apr 16 17:23:23.399844 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:23.399777    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3be2aca-401d-42fb-af06-e08e1ce0a152-kserve-provision-location\") pod \"b3be2aca-401d-42fb-af06-e08e1ce0a152\" (UID: \"b3be2aca-401d-42fb-af06-e08e1ce0a152\") "
Apr 16 17:23:23.400102 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:23.400077    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3be2aca-401d-42fb-af06-e08e1ce0a152-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3be2aca-401d-42fb-af06-e08e1ce0a152" (UID: "b3be2aca-401d-42fb-af06-e08e1ce0a152"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:23:23.500719 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:23.500694    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3be2aca-401d-42fb-af06-e08e1ce0a152-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:23:24.050154 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:24.050123    2578 generic.go:358] "Generic (PLEG): container finished" podID="b3be2aca-401d-42fb-af06-e08e1ce0a152" containerID="3469fd6778f1f99bf469a1370fee124a61a7580421f2441b87866ee2ad9aa1d5" exitCode=0
Apr 16 17:23:24.050542 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:24.050186    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f" event={"ID":"b3be2aca-401d-42fb-af06-e08e1ce0a152","Type":"ContainerDied","Data":"3469fd6778f1f99bf469a1370fee124a61a7580421f2441b87866ee2ad9aa1d5"}
Apr 16 17:23:24.050542 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:24.050207    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f"
Apr 16 17:23:24.050542 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:24.050220    2578 scope.go:117] "RemoveContainer" containerID="3469fd6778f1f99bf469a1370fee124a61a7580421f2441b87866ee2ad9aa1d5"
Apr 16 17:23:24.050542 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:24.050211    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f" event={"ID":"b3be2aca-401d-42fb-af06-e08e1ce0a152","Type":"ContainerDied","Data":"3c2f9772aa0d8607aee2f87cf039901180293ba94980b9839f650ccd8d475bb5"}
Apr 16 17:23:24.058098 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:24.058083    2578 scope.go:117] "RemoveContainer" containerID="327e31c95cae7ed1ec45a657c15bf265bcf4625f1e4c77cb3eb48c3e551d2cc4"
Apr 16 17:23:24.065000 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:24.064980    2578 scope.go:117] "RemoveContainer" containerID="3469fd6778f1f99bf469a1370fee124a61a7580421f2441b87866ee2ad9aa1d5"
Apr 16 17:23:24.065228 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:23:24.065211    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3469fd6778f1f99bf469a1370fee124a61a7580421f2441b87866ee2ad9aa1d5\": container with ID starting with 3469fd6778f1f99bf469a1370fee124a61a7580421f2441b87866ee2ad9aa1d5 not found: ID does not exist" containerID="3469fd6778f1f99bf469a1370fee124a61a7580421f2441b87866ee2ad9aa1d5"
Apr 16 17:23:24.065290 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:24.065234    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3469fd6778f1f99bf469a1370fee124a61a7580421f2441b87866ee2ad9aa1d5"} err="failed to get container status \"3469fd6778f1f99bf469a1370fee124a61a7580421f2441b87866ee2ad9aa1d5\": rpc error: code = NotFound desc = could not find container \"3469fd6778f1f99bf469a1370fee124a61a7580421f2441b87866ee2ad9aa1d5\": container with ID starting with 3469fd6778f1f99bf469a1370fee124a61a7580421f2441b87866ee2ad9aa1d5 not found: ID does not exist"
Apr 16 17:23:24.065290 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:24.065255    2578 scope.go:117] "RemoveContainer" containerID="327e31c95cae7ed1ec45a657c15bf265bcf4625f1e4c77cb3eb48c3e551d2cc4"
Apr 16 17:23:24.065457 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:23:24.065443    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"327e31c95cae7ed1ec45a657c15bf265bcf4625f1e4c77cb3eb48c3e551d2cc4\": container with ID starting with 327e31c95cae7ed1ec45a657c15bf265bcf4625f1e4c77cb3eb48c3e551d2cc4 not found: ID does not exist" containerID="327e31c95cae7ed1ec45a657c15bf265bcf4625f1e4c77cb3eb48c3e551d2cc4"
Apr 16 17:23:24.065494 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:24.065461    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"327e31c95cae7ed1ec45a657c15bf265bcf4625f1e4c77cb3eb48c3e551d2cc4"} err="failed to get container status \"327e31c95cae7ed1ec45a657c15bf265bcf4625f1e4c77cb3eb48c3e551d2cc4\": rpc error: code = NotFound desc = could not find container \"327e31c95cae7ed1ec45a657c15bf265bcf4625f1e4c77cb3eb48c3e551d2cc4\": container with ID starting with 327e31c95cae7ed1ec45a657c15bf265bcf4625f1e4c77cb3eb48c3e551d2cc4 not found: ID does not exist"
Apr 16 17:23:24.072136 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:24.072119    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f"]
Apr 16 17:23:24.076519 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:24.076499    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-695b7cc5c-pjr4f"]
Apr 16 17:23:25.234525 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:25.234478    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3be2aca-401d-42fb-af06-e08e1ce0a152" path="/var/lib/kubelet/pods/b3be2aca-401d-42fb-af06-e08e1ce0a152/volumes"
Apr 16 17:23:33.047194 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:33.047114    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb" podUID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Apr 16 17:23:43.047043 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:43.047003    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb" podUID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Apr 16 17:23:53.047283 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:23:53.047242    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb" podUID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Apr 16 17:24:03.047200 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:03.047146    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb" podUID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Apr 16 17:24:13.047442 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:13.047400    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb" podUID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Apr 16 17:24:23.047743 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:23.047714    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"
Apr 16 17:24:29.498047 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:29.498015    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"]
Apr 16 17:24:29.498423 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:29.498277    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb" podUID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" containerName="kserve-container" containerID="cri-o://7e4c46b0e2b3f4c716c4d22eca0f0aa561a0f4d3a26937a598e94cf02df3af57" gracePeriod=30
Apr 16 17:24:30.581435 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:30.581402    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv"]
Apr 16 17:24:30.581810 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:30.581755    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3be2aca-401d-42fb-af06-e08e1ce0a152" containerName="storage-initializer"
Apr 16 17:24:30.581810 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:30.581769    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3be2aca-401d-42fb-af06-e08e1ce0a152" containerName="storage-initializer"
Apr 16 17:24:30.581810 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:30.581781    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3be2aca-401d-42fb-af06-e08e1ce0a152" containerName="kserve-container"
Apr 16 17:24:30.581810 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:30.581786    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3be2aca-401d-42fb-af06-e08e1ce0a152" containerName="kserve-container"
Apr 16 17:24:30.581947 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:30.581832    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="b3be2aca-401d-42fb-af06-e08e1ce0a152" containerName="kserve-container"
Apr 16 17:24:30.584772 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:30.584756    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv"
Apr 16 17:24:30.592083 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:30.592060    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv"]
Apr 16 17:24:30.681021 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:30.680990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37bf9a13-c1d4-412b-ac70-e4a3af7de6f0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv\" (UID: \"37bf9a13-c1d4-412b-ac70-e4a3af7de6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv"
Apr 16 17:24:30.781825 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:30.781790    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37bf9a13-c1d4-412b-ac70-e4a3af7de6f0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv\" (UID: \"37bf9a13-c1d4-412b-ac70-e4a3af7de6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv"
Apr 16 17:24:30.782182 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:30.782163    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37bf9a13-c1d4-412b-ac70-e4a3af7de6f0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv\" (UID: \"37bf9a13-c1d4-412b-ac70-e4a3af7de6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv"
Apr 16 17:24:30.895712 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:30.895646    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv"
Apr 16 17:24:31.012016 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:31.011991    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv"]
Apr 16 17:24:31.014916 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:24:31.014890    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37bf9a13_c1d4_412b_ac70_e4a3af7de6f0.slice/crio-c0ede968df7b231f725663d352f13273e0c9724f1072820d16338a1360c221c0 WatchSource:0}: Error finding container c0ede968df7b231f725663d352f13273e0c9724f1072820d16338a1360c221c0: Status 404 returned error can't find the container with id c0ede968df7b231f725663d352f13273e0c9724f1072820d16338a1360c221c0
Apr 16 17:24:31.250125 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:31.250093    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv" event={"ID":"37bf9a13-c1d4-412b-ac70-e4a3af7de6f0","Type":"ContainerStarted","Data":"e4db0daacfdd7e137a81d006d980d9e6088803bf7989783cf42273445dcad457"}
Apr 16 17:24:31.250125 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:31.250123    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv" event={"ID":"37bf9a13-c1d4-412b-ac70-e4a3af7de6f0","Type":"ContainerStarted","Data":"c0ede968df7b231f725663d352f13273e0c9724f1072820d16338a1360c221c0"}
Apr 16 17:24:33.047676 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.047634    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb" podUID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Apr 16 17:24:33.222925 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.222901    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"
Apr 16 17:24:33.256894 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.256865    2578 generic.go:358] "Generic (PLEG): container finished" podID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" containerID="7e4c46b0e2b3f4c716c4d22eca0f0aa561a0f4d3a26937a598e94cf02df3af57" exitCode=0
Apr 16 17:24:33.257018 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.256940    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"
Apr 16 17:24:33.257018 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.256946    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb" event={"ID":"43cfa70b-8fc2-4271-80c2-7ceb6e264fa9","Type":"ContainerDied","Data":"7e4c46b0e2b3f4c716c4d22eca0f0aa561a0f4d3a26937a598e94cf02df3af57"}
Apr 16 17:24:33.257018 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.256982    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb" event={"ID":"43cfa70b-8fc2-4271-80c2-7ceb6e264fa9","Type":"ContainerDied","Data":"339d81b286449d885c21e201d485183af7565ad4eda26f015247e35665f5d517"}
Apr 16 17:24:33.257018 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.256997    2578 scope.go:117] "RemoveContainer" containerID="7e4c46b0e2b3f4c716c4d22eca0f0aa561a0f4d3a26937a598e94cf02df3af57"
Apr 16 17:24:33.264359 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.264341    2578 scope.go:117] "RemoveContainer" containerID="57812057cc2031a823a5a5aa8a5ac620874ba3bfc9440d179abe67d6872f3cbe"
Apr 16 17:24:33.270895 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.270879    2578 scope.go:117] "RemoveContainer" containerID="7e4c46b0e2b3f4c716c4d22eca0f0aa561a0f4d3a26937a598e94cf02df3af57"
Apr 16 17:24:33.271148 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:24:33.271128    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4c46b0e2b3f4c716c4d22eca0f0aa561a0f4d3a26937a598e94cf02df3af57\": container with ID starting with 7e4c46b0e2b3f4c716c4d22eca0f0aa561a0f4d3a26937a598e94cf02df3af57 not found: ID does not exist" containerID="7e4c46b0e2b3f4c716c4d22eca0f0aa561a0f4d3a26937a598e94cf02df3af57"
Apr 16 17:24:33.271214 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.271156    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4c46b0e2b3f4c716c4d22eca0f0aa561a0f4d3a26937a598e94cf02df3af57"} err="failed to get container status \"7e4c46b0e2b3f4c716c4d22eca0f0aa561a0f4d3a26937a598e94cf02df3af57\": rpc error: code = NotFound desc = could not find container \"7e4c46b0e2b3f4c716c4d22eca0f0aa561a0f4d3a26937a598e94cf02df3af57\": container with ID starting with 7e4c46b0e2b3f4c716c4d22eca0f0aa561a0f4d3a26937a598e94cf02df3af57 not found: ID does not exist"
Apr 16 17:24:33.271214 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.271172    2578 scope.go:117] "RemoveContainer" containerID="57812057cc2031a823a5a5aa8a5ac620874ba3bfc9440d179abe67d6872f3cbe"
Apr 16 17:24:33.271373 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:24:33.271355    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57812057cc2031a823a5a5aa8a5ac620874ba3bfc9440d179abe67d6872f3cbe\": container with ID starting with 57812057cc2031a823a5a5aa8a5ac620874ba3bfc9440d179abe67d6872f3cbe not found: ID does not exist" containerID="57812057cc2031a823a5a5aa8a5ac620874ba3bfc9440d179abe67d6872f3cbe"
Apr 16 17:24:33.271431 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.271382    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57812057cc2031a823a5a5aa8a5ac620874ba3bfc9440d179abe67d6872f3cbe"} err="failed to get container status \"57812057cc2031a823a5a5aa8a5ac620874ba3bfc9440d179abe67d6872f3cbe\": rpc error: code = NotFound desc = could not find container \"57812057cc2031a823a5a5aa8a5ac620874ba3bfc9440d179abe67d6872f3cbe\": container with ID starting with 57812057cc2031a823a5a5aa8a5ac620874ba3bfc9440d179abe67d6872f3cbe not found: ID does not exist"
Apr 16 17:24:33.301632 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.301589    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43cfa70b-8fc2-4271-80c2-7ceb6e264fa9-kserve-provision-location\") pod \"43cfa70b-8fc2-4271-80c2-7ceb6e264fa9\" (UID: \"43cfa70b-8fc2-4271-80c2-7ceb6e264fa9\") "
Apr 16 17:24:33.301632 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.301615    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43cfa70b-8fc2-4271-80c2-7ceb6e264fa9-cabundle-cert\") pod \"43cfa70b-8fc2-4271-80c2-7ceb6e264fa9\" (UID: \"43cfa70b-8fc2-4271-80c2-7ceb6e264fa9\") "
Apr 16 17:24:33.301844 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.301824    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cfa70b-8fc2-4271-80c2-7ceb6e264fa9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" (UID: "43cfa70b-8fc2-4271-80c2-7ceb6e264fa9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:24:33.302008 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.301989    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cfa70b-8fc2-4271-80c2-7ceb6e264fa9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" (UID: "43cfa70b-8fc2-4271-80c2-7ceb6e264fa9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 17:24:33.302477 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.302455    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43cfa70b-8fc2-4271-80c2-7ceb6e264fa9-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:24:33.302593 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.302481    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43cfa70b-8fc2-4271-80c2-7ceb6e264fa9-cabundle-cert\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:24:33.578399 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.578374    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"]
Apr 16 17:24:33.582359 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:33.582336    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5484987b89-zttbb"]
Apr 16 17:24:35.235015 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:35.234986    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" path="/var/lib/kubelet/pods/43cfa70b-8fc2-4271-80c2-7ceb6e264fa9/volumes"
Apr 16 17:24:37.269306 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:37.269282    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv_37bf9a13-c1d4-412b-ac70-e4a3af7de6f0/storage-initializer/0.log"
Apr 16 17:24:37.269646 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:37.269316    2578 generic.go:358] "Generic (PLEG): container finished" podID="37bf9a13-c1d4-412b-ac70-e4a3af7de6f0" containerID="e4db0daacfdd7e137a81d006d980d9e6088803bf7989783cf42273445dcad457" exitCode=1
Apr 16 17:24:37.269646 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:37.269341    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv" event={"ID":"37bf9a13-c1d4-412b-ac70-e4a3af7de6f0","Type":"ContainerDied","Data":"e4db0daacfdd7e137a81d006d980d9e6088803bf7989783cf42273445dcad457"}
Apr 16 17:24:38.273392 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:38.273365    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv_37bf9a13-c1d4-412b-ac70-e4a3af7de6f0/storage-initializer/0.log"
Apr 16 17:24:38.273769 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:38.273441    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv" event={"ID":"37bf9a13-c1d4-412b-ac70-e4a3af7de6f0","Type":"ContainerStarted","Data":"0848916792cde8af042c993bd8bf250a548aa4f1979a3bfbf55010b322b8a682"}
Apr 16 17:24:40.573130 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:40.573099    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv"]
Apr 16 17:24:40.573517 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:40.573379    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv" podUID="37bf9a13-c1d4-412b-ac70-e4a3af7de6f0" containerName="storage-initializer" containerID="cri-o://0848916792cde8af042c993bd8bf250a548aa4f1979a3bfbf55010b322b8a682" gracePeriod=30
Apr 16 17:24:41.509089 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.509070    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv_37bf9a13-c1d4-412b-ac70-e4a3af7de6f0/storage-initializer/1.log"
Apr 16 17:24:41.509430 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.509414    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv_37bf9a13-c1d4-412b-ac70-e4a3af7de6f0/storage-initializer/0.log"
Apr 16 17:24:41.509488 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.509477    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv"
Apr 16 17:24:41.658917 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.658860    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37bf9a13-c1d4-412b-ac70-e4a3af7de6f0-kserve-provision-location\") pod \"37bf9a13-c1d4-412b-ac70-e4a3af7de6f0\" (UID: \"37bf9a13-c1d4-412b-ac70-e4a3af7de6f0\") "
Apr 16 17:24:41.659210 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.659118    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bf9a13-c1d4-412b-ac70-e4a3af7de6f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37bf9a13-c1d4-412b-ac70-e4a3af7de6f0" (UID: "37bf9a13-c1d4-412b-ac70-e4a3af7de6f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:24:41.684759 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.684729    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"]
Apr 16 17:24:41.685087 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.685070    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37bf9a13-c1d4-412b-ac70-e4a3af7de6f0" containerName="storage-initializer"
Apr 16 17:24:41.685165 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.685089    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="37bf9a13-c1d4-412b-ac70-e4a3af7de6f0" containerName="storage-initializer"
Apr 16 17:24:41.685165 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.685100    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37bf9a13-c1d4-412b-ac70-e4a3af7de6f0" containerName="storage-initializer"
Apr 16 17:24:41.685165 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.685109    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="37bf9a13-c1d4-412b-ac70-e4a3af7de6f0" containerName="storage-initializer"
Apr 16 17:24:41.685165 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.685125    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" containerName="storage-initializer"
Apr 16 17:24:41.685165 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.685134    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" containerName="storage-initializer"
Apr 16 17:24:41.685165 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.685144    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" containerName="kserve-container"
Apr 16 17:24:41.685165 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.685152    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" containerName="kserve-container"
Apr 16 17:24:41.685517 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.685229    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="43cfa70b-8fc2-4271-80c2-7ceb6e264fa9" containerName="kserve-container"
Apr 16 17:24:41.685517 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.685245    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="37bf9a13-c1d4-412b-ac70-e4a3af7de6f0" containerName="storage-initializer"
Apr 16 17:24:41.685517 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.685385    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="37bf9a13-c1d4-412b-ac70-e4a3af7de6f0" containerName="storage-initializer"
Apr 16 17:24:41.688351 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.688333    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"
Apr 16 17:24:41.690628 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.690606    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Apr 16 17:24:41.696533 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.696512    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"]
Apr 16 17:24:41.760114 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.760089    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37bf9a13-c1d4-412b-ac70-e4a3af7de6f0-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:24:41.860372 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.860349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40475bdb-e92e-4af6-b655-326c8a6abc4d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75\" (UID: \"40475bdb-e92e-4af6-b655-326c8a6abc4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"
Apr 16 17:24:41.860473 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.860392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40475bdb-e92e-4af6-b655-326c8a6abc4d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75\" (UID: \"40475bdb-e92e-4af6-b655-326c8a6abc4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"
Apr 16 17:24:41.961170 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.961149    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40475bdb-e92e-4af6-b655-326c8a6abc4d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75\" (UID: \"40475bdb-e92e-4af6-b655-326c8a6abc4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"
Apr 16 17:24:41.961267 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.961180    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40475bdb-e92e-4af6-b655-326c8a6abc4d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75\" (UID: \"40475bdb-e92e-4af6-b655-326c8a6abc4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"
Apr 16 17:24:41.961491 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.961474    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40475bdb-e92e-4af6-b655-326c8a6abc4d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75\" (UID: \"40475bdb-e92e-4af6-b655-326c8a6abc4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"
Apr 16 17:24:41.961788 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.961770    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40475bdb-e92e-4af6-b655-326c8a6abc4d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75\" (UID: \"40475bdb-e92e-4af6-b655-326c8a6abc4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"
Apr 16 17:24:41.999324 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:41.999299    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"
Apr 16 17:24:42.114432 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.114407    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"]
Apr 16 17:24:42.116914 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:24:42.116883    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40475bdb_e92e_4af6_b655_326c8a6abc4d.slice/crio-10243214d72c8ce8da76706b9ca38bb528482e5da2408863132fd84c892e2c19 WatchSource:0}: Error finding container 10243214d72c8ce8da76706b9ca38bb528482e5da2408863132fd84c892e2c19: Status 404 returned error can't find the container with id 10243214d72c8ce8da76706b9ca38bb528482e5da2408863132fd84c892e2c19
Apr 16 17:24:42.285361 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.285283    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75" event={"ID":"40475bdb-e92e-4af6-b655-326c8a6abc4d","Type":"ContainerStarted","Data":"dd4452afc663fa1eec8b2437df9ff973a212732606c05a444e20626c112feb1b"}
Apr 16 17:24:42.285361 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.285320    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75" event={"ID":"40475bdb-e92e-4af6-b655-326c8a6abc4d","Type":"ContainerStarted","Data":"10243214d72c8ce8da76706b9ca38bb528482e5da2408863132fd84c892e2c19"}
Apr 16 17:24:42.286517 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.286498    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv_37bf9a13-c1d4-412b-ac70-e4a3af7de6f0/storage-initializer/1.log"
Apr 16 17:24:42.286869 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.286855    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv_37bf9a13-c1d4-412b-ac70-e4a3af7de6f0/storage-initializer/0.log"
Apr 16 17:24:42.286963 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.286887    2578 generic.go:358] "Generic (PLEG): container finished" podID="37bf9a13-c1d4-412b-ac70-e4a3af7de6f0" containerID="0848916792cde8af042c993bd8bf250a548aa4f1979a3bfbf55010b322b8a682" exitCode=1
Apr 16 17:24:42.286963 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.286916    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv" event={"ID":"37bf9a13-c1d4-412b-ac70-e4a3af7de6f0","Type":"ContainerDied","Data":"0848916792cde8af042c993bd8bf250a548aa4f1979a3bfbf55010b322b8a682"}
Apr 16 17:24:42.286963 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.286951    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv" event={"ID":"37bf9a13-c1d4-412b-ac70-e4a3af7de6f0","Type":"ContainerDied","Data":"c0ede968df7b231f725663d352f13273e0c9724f1072820d16338a1360c221c0"}
Apr 16 17:24:42.287132 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.286963    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv"
Apr 16 17:24:42.287132 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.286970    2578 scope.go:117] "RemoveContainer" containerID="0848916792cde8af042c993bd8bf250a548aa4f1979a3bfbf55010b322b8a682"
Apr 16 17:24:42.294674 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.294639    2578 scope.go:117] "RemoveContainer" containerID="e4db0daacfdd7e137a81d006d980d9e6088803bf7989783cf42273445dcad457"
Apr 16 17:24:42.303230 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.303212    2578 scope.go:117] "RemoveContainer" containerID="0848916792cde8af042c993bd8bf250a548aa4f1979a3bfbf55010b322b8a682"
Apr 16 17:24:42.303510 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:24:42.303492    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0848916792cde8af042c993bd8bf250a548aa4f1979a3bfbf55010b322b8a682\": container with ID starting with 0848916792cde8af042c993bd8bf250a548aa4f1979a3bfbf55010b322b8a682 not found: ID does not exist" containerID="0848916792cde8af042c993bd8bf250a548aa4f1979a3bfbf55010b322b8a682"
Apr 16 17:24:42.303670 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.303521    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0848916792cde8af042c993bd8bf250a548aa4f1979a3bfbf55010b322b8a682"} err="failed to get container status \"0848916792cde8af042c993bd8bf250a548aa4f1979a3bfbf55010b322b8a682\": rpc error: code = NotFound desc = could not find container \"0848916792cde8af042c993bd8bf250a548aa4f1979a3bfbf55010b322b8a682\": container with ID starting with 0848916792cde8af042c993bd8bf250a548aa4f1979a3bfbf55010b322b8a682 not found: ID does not exist"
Apr 16 17:24:42.303670 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.303542    2578 scope.go:117] "RemoveContainer" containerID="e4db0daacfdd7e137a81d006d980d9e6088803bf7989783cf42273445dcad457"
Apr 16 17:24:42.303832 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:24:42.303817    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4db0daacfdd7e137a81d006d980d9e6088803bf7989783cf42273445dcad457\": container with ID starting with e4db0daacfdd7e137a81d006d980d9e6088803bf7989783cf42273445dcad457 not found: ID does not exist" containerID="e4db0daacfdd7e137a81d006d980d9e6088803bf7989783cf42273445dcad457"
Apr 16 17:24:42.303909 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.303839    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4db0daacfdd7e137a81d006d980d9e6088803bf7989783cf42273445dcad457"} err="failed to get container status \"e4db0daacfdd7e137a81d006d980d9e6088803bf7989783cf42273445dcad457\": rpc error: code = NotFound desc = could not find container \"e4db0daacfdd7e137a81d006d980d9e6088803bf7989783cf42273445dcad457\": container with ID starting with e4db0daacfdd7e137a81d006d980d9e6088803bf7989783cf42273445dcad457 not found: ID does not exist"
Apr 16 17:24:42.323249 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.323227    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv"]
Apr 16 17:24:42.326529 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:42.326510    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-754959c6d7-s48pv"]
Apr 16 17:24:43.234880 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:43.234853    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37bf9a13-c1d4-412b-ac70-e4a3af7de6f0" path="/var/lib/kubelet/pods/37bf9a13-c1d4-412b-ac70-e4a3af7de6f0/volumes"
Apr 16 17:24:43.291399 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:43.291375    2578 generic.go:358] "Generic (PLEG): container finished" podID="40475bdb-e92e-4af6-b655-326c8a6abc4d" containerID="dd4452afc663fa1eec8b2437df9ff973a212732606c05a444e20626c112feb1b" exitCode=0
Apr 16 17:24:43.291515 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:43.291431    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75" event={"ID":"40475bdb-e92e-4af6-b655-326c8a6abc4d","Type":"ContainerDied","Data":"dd4452afc663fa1eec8b2437df9ff973a212732606c05a444e20626c112feb1b"}
Apr 16 17:24:44.295502 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:44.295460    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75" event={"ID":"40475bdb-e92e-4af6-b655-326c8a6abc4d","Type":"ContainerStarted","Data":"5873c91214620404e6dab10c14d4dc48cab3527262bd0d914cf569da4262db69"}
Apr 16 17:24:44.295883 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:44.295670    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"
Apr 16 17:24:44.296903 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:44.296879    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75" podUID="40475bdb-e92e-4af6-b655-326c8a6abc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Apr 16 17:24:44.312130 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:44.312089    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75" podStartSLOduration=3.312077859 podStartE2EDuration="3.312077859s" podCreationTimestamp="2026-04-16 17:24:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:24:44.310666004 +0000 UTC m=+3293.707882393" watchObservedRunningTime="2026-04-16 17:24:44.312077859 +0000 UTC m=+3293.709294318"
Apr 16 17:24:45.299269 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:45.299233    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75" podUID="40475bdb-e92e-4af6-b655-326c8a6abc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Apr 16 17:24:55.299940 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:24:55.299904    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75" podUID="40475bdb-e92e-4af6-b655-326c8a6abc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Apr 16 17:25:05.300009 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:05.299964    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75" podUID="40475bdb-e92e-4af6-b655-326c8a6abc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Apr 16 17:25:15.299461 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:15.299420    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75" podUID="40475bdb-e92e-4af6-b655-326c8a6abc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Apr 16 17:25:25.299486 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:25.299446    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75" podUID="40475bdb-e92e-4af6-b655-326c8a6abc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Apr 16 17:25:35.299461 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:35.299420    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75" podUID="40475bdb-e92e-4af6-b655-326c8a6abc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Apr 16 17:25:44.742782 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:44.742755    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 17:25:44.745345 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:44.745313    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 17:25:45.300698 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:45.300673    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"
Apr 16 17:25:51.760348 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:51.760320    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"]
Apr 16 17:25:51.760732 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:51.760613    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75" podUID="40475bdb-e92e-4af6-b655-326c8a6abc4d" containerName="kserve-container" containerID="cri-o://5873c91214620404e6dab10c14d4dc48cab3527262bd0d914cf569da4262db69" gracePeriod=30
Apr 16 17:25:52.768728 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:52.768693    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq"]
Apr 16 17:25:52.771871 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:52.771850    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq"
Apr 16 17:25:52.813571 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:52.813535    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq"]
Apr 16 17:25:52.872252 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:52.872229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f14adffd-537b-43f1-99e8-241586157743-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq\" (UID: \"f14adffd-537b-43f1-99e8-241586157743\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq"
Apr 16 17:25:52.973150 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:52.973123    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f14adffd-537b-43f1-99e8-241586157743-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq\" (UID: \"f14adffd-537b-43f1-99e8-241586157743\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq"
Apr 16 17:25:52.973443 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:52.973427    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f14adffd-537b-43f1-99e8-241586157743-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq\" (UID: \"f14adffd-537b-43f1-99e8-241586157743\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq"
Apr 16 17:25:53.081785 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:53.081730    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq"
Apr 16 17:25:53.194032 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:53.194002    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq"]
Apr 16 17:25:53.198196 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:25:53.198174    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14adffd_537b_43f1_99e8_241586157743.slice/crio-d3b0f8df880d9d80e115151354329f82d1b2e723b54c08f85710dc604d3f9086 WatchSource:0}: Error finding container d3b0f8df880d9d80e115151354329f82d1b2e723b54c08f85710dc604d3f9086: Status 404 returned error can't find the container with id d3b0f8df880d9d80e115151354329f82d1b2e723b54c08f85710dc604d3f9086
Apr 16 17:25:53.485934 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:53.485900    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq" event={"ID":"f14adffd-537b-43f1-99e8-241586157743","Type":"ContainerStarted","Data":"45f4844df152dc04d8a1a1598cdc5a5358d912f26f040b71d40d0b48daae6f75"}
Apr 16 17:25:53.486094 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:53.485940    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq" event={"ID":"f14adffd-537b-43f1-99e8-241586157743","Type":"ContainerStarted","Data":"d3b0f8df880d9d80e115151354329f82d1b2e723b54c08f85710dc604d3f9086"}
Apr 16 17:25:55.300181 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:55.300143    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75" podUID="40475bdb-e92e-4af6-b655-326c8a6abc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Apr 16 17:25:55.493819 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:55.493783    2578 generic.go:358] "Generic (PLEG): container finished" podID="40475bdb-e92e-4af6-b655-326c8a6abc4d" containerID="5873c91214620404e6dab10c14d4dc48cab3527262bd0d914cf569da4262db69" exitCode=0
Apr 16 17:25:55.493943 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:55.493850    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75" event={"ID":"40475bdb-e92e-4af6-b655-326c8a6abc4d","Type":"ContainerDied","Data":"5873c91214620404e6dab10c14d4dc48cab3527262bd0d914cf569da4262db69"}
Apr 16 17:25:55.493943 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:55.493899    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75" event={"ID":"40475bdb-e92e-4af6-b655-326c8a6abc4d","Type":"ContainerDied","Data":"10243214d72c8ce8da76706b9ca38bb528482e5da2408863132fd84c892e2c19"}
Apr 16 17:25:55.493943 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:55.493917    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10243214d72c8ce8da76706b9ca38bb528482e5da2408863132fd84c892e2c19"
Apr 16 17:25:55.495270 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:55.495252    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq_f14adffd-537b-43f1-99e8-241586157743/storage-initializer/0.log"
Apr 16 17:25:55.495398 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:55.495286    2578 generic.go:358] "Generic (PLEG): container finished" podID="f14adffd-537b-43f1-99e8-241586157743" containerID="45f4844df152dc04d8a1a1598cdc5a5358d912f26f040b71d40d0b48daae6f75" exitCode=1
Apr 16 17:25:55.495398 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:55.495311    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq" event={"ID":"f14adffd-537b-43f1-99e8-241586157743","Type":"ContainerDied","Data":"45f4844df152dc04d8a1a1598cdc5a5358d912f26f040b71d40d0b48daae6f75"}
Apr 16 17:25:55.504431 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:55.504408    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"
Apr 16 17:25:55.591992 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:55.591937    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40475bdb-e92e-4af6-b655-326c8a6abc4d-cabundle-cert\") pod \"40475bdb-e92e-4af6-b655-326c8a6abc4d\" (UID: \"40475bdb-e92e-4af6-b655-326c8a6abc4d\") "
Apr 16 17:25:55.592113 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:55.591997    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40475bdb-e92e-4af6-b655-326c8a6abc4d-kserve-provision-location\") pod \"40475bdb-e92e-4af6-b655-326c8a6abc4d\" (UID: \"40475bdb-e92e-4af6-b655-326c8a6abc4d\") "
Apr 16 17:25:55.592323 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:55.592298    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40475bdb-e92e-4af6-b655-326c8a6abc4d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "40475bdb-e92e-4af6-b655-326c8a6abc4d" (UID: "40475bdb-e92e-4af6-b655-326c8a6abc4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 17:25:55.592323 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:55.592308    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40475bdb-e92e-4af6-b655-326c8a6abc4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40475bdb-e92e-4af6-b655-326c8a6abc4d" (UID: "40475bdb-e92e-4af6-b655-326c8a6abc4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:25:55.693037 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:55.693013    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40475bdb-e92e-4af6-b655-326c8a6abc4d-cabundle-cert\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:25:55.693037 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:55.693034    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40475bdb-e92e-4af6-b655-326c8a6abc4d-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:25:56.499710 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:56.499688    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq_f14adffd-537b-43f1-99e8-241586157743/storage-initializer/0.log"
Apr 16 17:25:56.500097 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:56.499799    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq" event={"ID":"f14adffd-537b-43f1-99e8-241586157743","Type":"ContainerStarted","Data":"48cfbc7e32d2c281570f21ebdd5a5cf30ae0b3799bad348a76894eb61cbde34d"}
Apr 16 17:25:56.500097 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:56.499829    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"
Apr 16 17:25:56.533485 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:56.533459    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"]
Apr 16 17:25:56.539805 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:56.539784    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-566f78cd49-wrs75"]
Apr 16 17:25:57.234947 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:25:57.234919    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40475bdb-e92e-4af6-b655-326c8a6abc4d" path="/var/lib/kubelet/pods/40475bdb-e92e-4af6-b655-326c8a6abc4d/volumes"
Apr 16 17:26:01.515050 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:01.515024    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq_f14adffd-537b-43f1-99e8-241586157743/storage-initializer/1.log"
Apr 16 17:26:01.515408 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:01.515364    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq_f14adffd-537b-43f1-99e8-241586157743/storage-initializer/0.log"
Apr 16 17:26:01.515408 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:01.515394    2578 generic.go:358] "Generic (PLEG): container finished" podID="f14adffd-537b-43f1-99e8-241586157743" containerID="48cfbc7e32d2c281570f21ebdd5a5cf30ae0b3799bad348a76894eb61cbde34d" exitCode=1
Apr 16 17:26:01.515519 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:01.515424    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq" event={"ID":"f14adffd-537b-43f1-99e8-241586157743","Type":"ContainerDied","Data":"48cfbc7e32d2c281570f21ebdd5a5cf30ae0b3799bad348a76894eb61cbde34d"}
Apr 16 17:26:01.515519 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:01.515451    2578 scope.go:117] "RemoveContainer" containerID="45f4844df152dc04d8a1a1598cdc5a5358d912f26f040b71d40d0b48daae6f75"
Apr 16 17:26:01.515806 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:01.515784    2578 scope.go:117] "RemoveContainer" containerID="45f4844df152dc04d8a1a1598cdc5a5358d912f26f040b71d40d0b48daae6f75"
Apr 16 17:26:01.525481 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:26:01.525454    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq_kserve-ci-e2e-test_f14adffd-537b-43f1-99e8-241586157743_0 in pod sandbox d3b0f8df880d9d80e115151354329f82d1b2e723b54c08f85710dc604d3f9086 from index: no such id: '45f4844df152dc04d8a1a1598cdc5a5358d912f26f040b71d40d0b48daae6f75'" containerID="45f4844df152dc04d8a1a1598cdc5a5358d912f26f040b71d40d0b48daae6f75"
Apr 16 17:26:01.525586 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:26:01.525501    2578 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq_kserve-ci-e2e-test_f14adffd-537b-43f1-99e8-241586157743_0 in pod sandbox d3b0f8df880d9d80e115151354329f82d1b2e723b54c08f85710dc604d3f9086 from index: no such id: '45f4844df152dc04d8a1a1598cdc5a5358d912f26f040b71d40d0b48daae6f75'; Skipping pod \"isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq_kserve-ci-e2e-test(f14adffd-537b-43f1-99e8-241586157743)\"" logger="UnhandledError"
Apr 16 17:26:01.526809 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:26:01.526789    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq_kserve-ci-e2e-test(f14adffd-537b-43f1-99e8-241586157743)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq" podUID="f14adffd-537b-43f1-99e8-241586157743"
Apr 16 17:26:02.520541 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:02.520513    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq_f14adffd-537b-43f1-99e8-241586157743/storage-initializer/1.log"
Apr 16 17:26:02.780437 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:02.780366    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq"]
Apr 16 17:26:02.900917 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:02.900892    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq_f14adffd-537b-43f1-99e8-241586157743/storage-initializer/1.log"
Apr 16 17:26:02.901000 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:02.900959    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq"
Apr 16 17:26:03.047897 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.047845    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f14adffd-537b-43f1-99e8-241586157743-kserve-provision-location\") pod \"f14adffd-537b-43f1-99e8-241586157743\" (UID: \"f14adffd-537b-43f1-99e8-241586157743\") "
Apr 16 17:26:03.048100 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.048080    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14adffd-537b-43f1-99e8-241586157743-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f14adffd-537b-43f1-99e8-241586157743" (UID: "f14adffd-537b-43f1-99e8-241586157743"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:26:03.148905 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.148877    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f14adffd-537b-43f1-99e8-241586157743-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:26:03.524908 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.524879    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq_f14adffd-537b-43f1-99e8-241586157743/storage-initializer/1.log"
Apr 16 17:26:03.525234 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.524938    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq" event={"ID":"f14adffd-537b-43f1-99e8-241586157743","Type":"ContainerDied","Data":"d3b0f8df880d9d80e115151354329f82d1b2e723b54c08f85710dc604d3f9086"}
Apr 16 17:26:03.525234 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.524969    2578 scope.go:117] "RemoveContainer" containerID="48cfbc7e32d2c281570f21ebdd5a5cf30ae0b3799bad348a76894eb61cbde34d"
Apr 16 17:26:03.525234 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.524987    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq"
Apr 16 17:26:03.554660 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.554634    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq"]
Apr 16 17:26:03.559795 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.559768    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-665c965d55-kgpsq"]
Apr 16 17:26:03.828866 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.828792    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"]
Apr 16 17:26:03.829093 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.829081    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f14adffd-537b-43f1-99e8-241586157743" containerName="storage-initializer"
Apr 16 17:26:03.829142 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.829094    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f14adffd-537b-43f1-99e8-241586157743" containerName="storage-initializer"
Apr 16 17:26:03.829142 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.829108    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40475bdb-e92e-4af6-b655-326c8a6abc4d" containerName="storage-initializer"
Apr 16 17:26:03.829142 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.829114    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="40475bdb-e92e-4af6-b655-326c8a6abc4d" containerName="storage-initializer"
Apr 16 17:26:03.829142 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.829130    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40475bdb-e92e-4af6-b655-326c8a6abc4d" containerName="kserve-container"
Apr 16 17:26:03.829142 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.829136    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="40475bdb-e92e-4af6-b655-326c8a6abc4d" containerName="kserve-container"
Apr 16 17:26:03.829313 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.829179    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="f14adffd-537b-43f1-99e8-241586157743" containerName="storage-initializer"
Apr 16 17:26:03.829313 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.829187    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="f14adffd-537b-43f1-99e8-241586157743" containerName="storage-initializer"
Apr 16 17:26:03.829313 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.829195    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="40475bdb-e92e-4af6-b655-326c8a6abc4d" containerName="kserve-container"
Apr 16 17:26:03.829313 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.829239    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f14adffd-537b-43f1-99e8-241586157743" containerName="storage-initializer"
Apr 16 17:26:03.829313 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.829244    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f14adffd-537b-43f1-99e8-241586157743" containerName="storage-initializer"
Apr 16 17:26:03.833256 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.833240    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"
Apr 16 17:26:03.835660 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.835628    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Apr 16 17:26:03.835798 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.835743    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fhlr7\""
Apr 16 17:26:03.835875 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.835812    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Apr 16 17:26:03.839272 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.839249    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"]
Apr 16 17:26:03.955997 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.955973    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6023fb09-4f85-4f11-81cc-ed000af731bc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986\" (UID: \"6023fb09-4f85-4f11-81cc-ed000af731bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"
Apr 16 17:26:03.956098 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:03.956062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6023fb09-4f85-4f11-81cc-ed000af731bc-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986\" (UID: \"6023fb09-4f85-4f11-81cc-ed000af731bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"
Apr 16 17:26:04.056370 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:04.056348    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6023fb09-4f85-4f11-81cc-ed000af731bc-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986\" (UID: \"6023fb09-4f85-4f11-81cc-ed000af731bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"
Apr 16 17:26:04.056452 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:04.056386    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6023fb09-4f85-4f11-81cc-ed000af731bc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986\" (UID: \"6023fb09-4f85-4f11-81cc-ed000af731bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"
Apr 16 17:26:04.056704 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:04.056689    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6023fb09-4f85-4f11-81cc-ed000af731bc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986\" (UID: \"6023fb09-4f85-4f11-81cc-ed000af731bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"
Apr 16 17:26:04.056940 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:04.056923    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6023fb09-4f85-4f11-81cc-ed000af731bc-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986\" (UID: \"6023fb09-4f85-4f11-81cc-ed000af731bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"
Apr 16 17:26:04.143303 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:04.143236    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"
Apr 16 17:26:04.255377 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:04.255353    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"]
Apr 16 17:26:04.258029 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:26:04.258002    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6023fb09_4f85_4f11_81cc_ed000af731bc.slice/crio-a5488f1955a085fc143b0b6cfa466c97c006dddc6b9da5c156835c00a385a20b WatchSource:0}: Error finding container a5488f1955a085fc143b0b6cfa466c97c006dddc6b9da5c156835c00a385a20b: Status 404 returned error can't find the container with id a5488f1955a085fc143b0b6cfa466c97c006dddc6b9da5c156835c00a385a20b
Apr 16 17:26:04.529898 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:04.529867    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986" event={"ID":"6023fb09-4f85-4f11-81cc-ed000af731bc","Type":"ContainerStarted","Data":"93aaab72e2c998f159a36a42fdb7ad79ed4ee95d0e1c7bdccc259da584d2df5a"}
Apr 16 17:26:04.530294 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:04.529905    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986" event={"ID":"6023fb09-4f85-4f11-81cc-ed000af731bc","Type":"ContainerStarted","Data":"a5488f1955a085fc143b0b6cfa466c97c006dddc6b9da5c156835c00a385a20b"}
Apr 16 17:26:05.235283 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:05.235256    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f14adffd-537b-43f1-99e8-241586157743" path="/var/lib/kubelet/pods/f14adffd-537b-43f1-99e8-241586157743/volumes"
Apr 16 17:26:05.533644 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:05.533616    2578 generic.go:358] "Generic (PLEG): container finished" podID="6023fb09-4f85-4f11-81cc-ed000af731bc" containerID="93aaab72e2c998f159a36a42fdb7ad79ed4ee95d0e1c7bdccc259da584d2df5a" exitCode=0
Apr 16 17:26:05.534056 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:05.533700    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986" event={"ID":"6023fb09-4f85-4f11-81cc-ed000af731bc","Type":"ContainerDied","Data":"93aaab72e2c998f159a36a42fdb7ad79ed4ee95d0e1c7bdccc259da584d2df5a"}
Apr 16 17:26:06.538186 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:06.538159    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986" event={"ID":"6023fb09-4f85-4f11-81cc-ed000af731bc","Type":"ContainerStarted","Data":"d2a280f7b0526bcc317255ed0e5f44361b78a451fd5dde35333d5ec0ce6295fd"}
Apr 16 17:26:06.538651 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:06.538399    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"
Apr 16 17:26:06.539592 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:06.539546    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986" podUID="6023fb09-4f85-4f11-81cc-ed000af731bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Apr 16 17:26:06.556250 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:06.556213    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986" podStartSLOduration=3.556202742 podStartE2EDuration="3.556202742s" podCreationTimestamp="2026-04-16 17:26:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:26:06.554673238 +0000 UTC m=+3375.951889625" watchObservedRunningTime="2026-04-16 17:26:06.556202742 +0000 UTC m=+3375.953419131"
Apr 16 17:26:07.541527 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:07.541486    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986" podUID="6023fb09-4f85-4f11-81cc-ed000af731bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Apr 16 17:26:17.541484 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:17.541442    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986" podUID="6023fb09-4f85-4f11-81cc-ed000af731bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Apr 16 17:26:27.542248 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:27.542205    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986" podUID="6023fb09-4f85-4f11-81cc-ed000af731bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Apr 16 17:26:37.541652 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:37.541574    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986" podUID="6023fb09-4f85-4f11-81cc-ed000af731bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Apr 16 17:26:47.542344 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:47.542300    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986" podUID="6023fb09-4f85-4f11-81cc-ed000af731bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Apr 16 17:26:57.541784 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:26:57.541742    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986" podUID="6023fb09-4f85-4f11-81cc-ed000af731bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Apr 16 17:27:07.543264 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:07.543230    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"
Apr 16 17:27:13.864592 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:13.864536    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"]
Apr 16 17:27:13.865118 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:13.864806    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986" podUID="6023fb09-4f85-4f11-81cc-ed000af731bc" containerName="kserve-container" containerID="cri-o://d2a280f7b0526bcc317255ed0e5f44361b78a451fd5dde35333d5ec0ce6295fd" gracePeriod=30
Apr 16 17:27:14.929679 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:14.929644    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh"]
Apr 16 17:27:14.932449 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:14.932427    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh"
Apr 16 17:27:14.942715 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:14.942692    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh"]
Apr 16 17:27:15.031192 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:15.031164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432c565c-ee80-4f78-9c40-d60b9a515487-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh\" (UID: \"432c565c-ee80-4f78-9c40-d60b9a515487\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh"
Apr 16 17:27:15.131485 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:15.131449    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432c565c-ee80-4f78-9c40-d60b9a515487-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh\" (UID: \"432c565c-ee80-4f78-9c40-d60b9a515487\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh"
Apr 16 17:27:15.131825 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:15.131804    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432c565c-ee80-4f78-9c40-d60b9a515487-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh\" (UID: \"432c565c-ee80-4f78-9c40-d60b9a515487\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh"
Apr 16 17:27:15.242922 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:15.242903    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh"
Apr 16 17:27:15.353825 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:15.353790    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh"]
Apr 16 17:27:15.356497 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:27:15.356470    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432c565c_ee80_4f78_9c40_d60b9a515487.slice/crio-a865def94b7c90b31a391bb0e002bffdc08d86bcac73f7e21590dd2a4f500000 WatchSource:0}: Error finding container a865def94b7c90b31a391bb0e002bffdc08d86bcac73f7e21590dd2a4f500000: Status 404 returned error can't find the container with id a865def94b7c90b31a391bb0e002bffdc08d86bcac73f7e21590dd2a4f500000
Apr 16 17:27:15.358371 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:15.358353    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 17:27:15.736796 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:15.736752    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh" event={"ID":"432c565c-ee80-4f78-9c40-d60b9a515487","Type":"ContainerStarted","Data":"6ff215ba394ceb90b3868093d1ee00a4ed5d53cee4f3ac5330cba5c021460483"}
Apr 16 17:27:15.736796 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:15.736797    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh" event={"ID":"432c565c-ee80-4f78-9c40-d60b9a515487","Type":"ContainerStarted","Data":"a865def94b7c90b31a391bb0e002bffdc08d86bcac73f7e21590dd2a4f500000"}
Apr 16 17:27:17.591154 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.591131    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"
Apr 16 17:27:17.651541 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.651519    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6023fb09-4f85-4f11-81cc-ed000af731bc-kserve-provision-location\") pod \"6023fb09-4f85-4f11-81cc-ed000af731bc\" (UID: \"6023fb09-4f85-4f11-81cc-ed000af731bc\") "
Apr 16 17:27:17.651664 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.651547    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6023fb09-4f85-4f11-81cc-ed000af731bc-cabundle-cert\") pod \"6023fb09-4f85-4f11-81cc-ed000af731bc\" (UID: \"6023fb09-4f85-4f11-81cc-ed000af731bc\") "
Apr 16 17:27:17.651850 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.651828    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6023fb09-4f85-4f11-81cc-ed000af731bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6023fb09-4f85-4f11-81cc-ed000af731bc" (UID: "6023fb09-4f85-4f11-81cc-ed000af731bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:27:17.651931 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.651915    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6023fb09-4f85-4f11-81cc-ed000af731bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6023fb09-4f85-4f11-81cc-ed000af731bc" (UID: "6023fb09-4f85-4f11-81cc-ed000af731bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 17:27:17.744032 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.744001    2578 generic.go:358] "Generic (PLEG): container finished" podID="6023fb09-4f85-4f11-81cc-ed000af731bc" containerID="d2a280f7b0526bcc317255ed0e5f44361b78a451fd5dde35333d5ec0ce6295fd" exitCode=0
Apr 16 17:27:17.744142 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.744074    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"
Apr 16 17:27:17.744142 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.744086    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986" event={"ID":"6023fb09-4f85-4f11-81cc-ed000af731bc","Type":"ContainerDied","Data":"d2a280f7b0526bcc317255ed0e5f44361b78a451fd5dde35333d5ec0ce6295fd"}
Apr 16 17:27:17.744142 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.744121    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986" event={"ID":"6023fb09-4f85-4f11-81cc-ed000af731bc","Type":"ContainerDied","Data":"a5488f1955a085fc143b0b6cfa466c97c006dddc6b9da5c156835c00a385a20b"}
Apr 16 17:27:17.744142 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.744138    2578 scope.go:117] "RemoveContainer" containerID="d2a280f7b0526bcc317255ed0e5f44361b78a451fd5dde35333d5ec0ce6295fd"
Apr 16 17:27:17.752054 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.752034    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6023fb09-4f85-4f11-81cc-ed000af731bc-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:27:17.752145 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.752055    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6023fb09-4f85-4f11-81cc-ed000af731bc-cabundle-cert\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:27:17.752357 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.752344    2578 scope.go:117] "RemoveContainer" containerID="93aaab72e2c998f159a36a42fdb7ad79ed4ee95d0e1c7bdccc259da584d2df5a"
Apr 16 17:27:17.759081 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.759062    2578 scope.go:117] "RemoveContainer" containerID="d2a280f7b0526bcc317255ed0e5f44361b78a451fd5dde35333d5ec0ce6295fd"
Apr 16 17:27:17.759289 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:27:17.759273    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a280f7b0526bcc317255ed0e5f44361b78a451fd5dde35333d5ec0ce6295fd\": container with ID starting with d2a280f7b0526bcc317255ed0e5f44361b78a451fd5dde35333d5ec0ce6295fd not found: ID does not exist" containerID="d2a280f7b0526bcc317255ed0e5f44361b78a451fd5dde35333d5ec0ce6295fd"
Apr 16 17:27:17.759337 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.759297    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a280f7b0526bcc317255ed0e5f44361b78a451fd5dde35333d5ec0ce6295fd"} err="failed to get container status \"d2a280f7b0526bcc317255ed0e5f44361b78a451fd5dde35333d5ec0ce6295fd\": rpc error: code = NotFound desc = could not find container \"d2a280f7b0526bcc317255ed0e5f44361b78a451fd5dde35333d5ec0ce6295fd\": container with ID starting with d2a280f7b0526bcc317255ed0e5f44361b78a451fd5dde35333d5ec0ce6295fd not found: ID does not exist"
Apr 16 17:27:17.759337 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.759313    2578 scope.go:117] "RemoveContainer" containerID="93aaab72e2c998f159a36a42fdb7ad79ed4ee95d0e1c7bdccc259da584d2df5a"
Apr 16 17:27:17.759506 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:27:17.759490    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93aaab72e2c998f159a36a42fdb7ad79ed4ee95d0e1c7bdccc259da584d2df5a\": container with ID starting with 93aaab72e2c998f159a36a42fdb7ad79ed4ee95d0e1c7bdccc259da584d2df5a not found: ID does not exist" containerID="93aaab72e2c998f159a36a42fdb7ad79ed4ee95d0e1c7bdccc259da584d2df5a"
Apr 16 17:27:17.759546 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.759510    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93aaab72e2c998f159a36a42fdb7ad79ed4ee95d0e1c7bdccc259da584d2df5a"} err="failed to get container status \"93aaab72e2c998f159a36a42fdb7ad79ed4ee95d0e1c7bdccc259da584d2df5a\": rpc error: code = NotFound desc = could not find container \"93aaab72e2c998f159a36a42fdb7ad79ed4ee95d0e1c7bdccc259da584d2df5a\": container with ID starting with 93aaab72e2c998f159a36a42fdb7ad79ed4ee95d0e1c7bdccc259da584d2df5a not found: ID does not exist"
Apr 16 17:27:17.763199 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.763178    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"]
Apr 16 17:27:17.766456 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:17.766436    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986"]
Apr 16 17:27:18.542101 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:18.542065    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5f684c44cb-xn986" podUID="6023fb09-4f85-4f11-81cc-ed000af731bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: i/o timeout"
Apr 16 17:27:19.234939 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:19.234906    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6023fb09-4f85-4f11-81cc-ed000af731bc" path="/var/lib/kubelet/pods/6023fb09-4f85-4f11-81cc-ed000af731bc/volumes"
Apr 16 17:27:20.756115 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:20.756094    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh_432c565c-ee80-4f78-9c40-d60b9a515487/storage-initializer/0.log"
Apr 16 17:27:20.756376 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:20.756129    2578 generic.go:358] "Generic (PLEG): container finished" podID="432c565c-ee80-4f78-9c40-d60b9a515487" containerID="6ff215ba394ceb90b3868093d1ee00a4ed5d53cee4f3ac5330cba5c021460483" exitCode=1
Apr 16 17:27:20.756376 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:20.756195    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh" event={"ID":"432c565c-ee80-4f78-9c40-d60b9a515487","Type":"ContainerDied","Data":"6ff215ba394ceb90b3868093d1ee00a4ed5d53cee4f3ac5330cba5c021460483"}
Apr 16 17:27:21.761004 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:21.760976    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh_432c565c-ee80-4f78-9c40-d60b9a515487/storage-initializer/0.log"
Apr 16 17:27:21.761374 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:21.761039    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh" event={"ID":"432c565c-ee80-4f78-9c40-d60b9a515487","Type":"ContainerStarted","Data":"a7e001a58e0c30838a7e5829a0b6beff57ec3cedebcf19953fc6b49946dc6493"}
Apr 16 17:27:24.937940 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:24.937900    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh"]
Apr 16 17:27:24.938304 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:24.938135    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh" podUID="432c565c-ee80-4f78-9c40-d60b9a515487" containerName="storage-initializer" containerID="cri-o://a7e001a58e0c30838a7e5829a0b6beff57ec3cedebcf19953fc6b49946dc6493" gracePeriod=30
Apr 16 17:27:27.170304 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.170279    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh_432c565c-ee80-4f78-9c40-d60b9a515487/storage-initializer/1.log"
Apr 16 17:27:27.170632 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.170619    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh_432c565c-ee80-4f78-9c40-d60b9a515487/storage-initializer/0.log"
Apr 16 17:27:27.170693 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.170682    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh"
Apr 16 17:27:27.221786 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.221764    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432c565c-ee80-4f78-9c40-d60b9a515487-kserve-provision-location\") pod \"432c565c-ee80-4f78-9c40-d60b9a515487\" (UID: \"432c565c-ee80-4f78-9c40-d60b9a515487\") "
Apr 16 17:27:27.222004 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.221985    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432c565c-ee80-4f78-9c40-d60b9a515487-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "432c565c-ee80-4f78-9c40-d60b9a515487" (UID: "432c565c-ee80-4f78-9c40-d60b9a515487"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:27:27.322427 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.322404    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432c565c-ee80-4f78-9c40-d60b9a515487-kserve-provision-location\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:27:27.700047 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.700022    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z2hj7/must-gather-p265v"]
Apr 16 17:27:27.700315 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.700303    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432c565c-ee80-4f78-9c40-d60b9a515487" containerName="storage-initializer"
Apr 16 17:27:27.700359 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.700316    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="432c565c-ee80-4f78-9c40-d60b9a515487" containerName="storage-initializer"
Apr 16 17:27:27.700359 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.700326    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6023fb09-4f85-4f11-81cc-ed000af731bc" containerName="kserve-container"
Apr 16 17:27:27.700359 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.700332    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6023fb09-4f85-4f11-81cc-ed000af731bc" containerName="kserve-container"
Apr 16 17:27:27.700359 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.700341    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6023fb09-4f85-4f11-81cc-ed000af731bc" containerName="storage-initializer"
Apr 16 17:27:27.700359 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.700346    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6023fb09-4f85-4f11-81cc-ed000af731bc" containerName="storage-initializer"
Apr 16 17:27:27.700359 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.700359    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432c565c-ee80-4f78-9c40-d60b9a515487" containerName="storage-initializer"
Apr 16 17:27:27.700549 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.700365    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="432c565c-ee80-4f78-9c40-d60b9a515487" containerName="storage-initializer"
Apr 16 17:27:27.700549 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.700409    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="432c565c-ee80-4f78-9c40-d60b9a515487" containerName="storage-initializer"
Apr 16 17:27:27.700549 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.700418    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="6023fb09-4f85-4f11-81cc-ed000af731bc" containerName="kserve-container"
Apr 16 17:27:27.700549 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.700427    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="432c565c-ee80-4f78-9c40-d60b9a515487" containerName="storage-initializer"
Apr 16 17:27:27.703381 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.703364    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z2hj7/must-gather-p265v"
Apr 16 17:27:27.705692 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.705665    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z2hj7\"/\"openshift-service-ca.crt\""
Apr 16 17:27:27.705798 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.705774    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z2hj7\"/\"default-dockercfg-v8rmn\""
Apr 16 17:27:27.705865 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.705853    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z2hj7\"/\"kube-root-ca.crt\""
Apr 16 17:27:27.710442 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.710349    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z2hj7/must-gather-p265v"]
Apr 16 17:27:27.778936 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.778919    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh_432c565c-ee80-4f78-9c40-d60b9a515487/storage-initializer/1.log"
Apr 16 17:27:27.779256 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.779242    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh_432c565c-ee80-4f78-9c40-d60b9a515487/storage-initializer/0.log"
Apr 16 17:27:27.779339 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.779274    2578 generic.go:358] "Generic (PLEG): container finished" podID="432c565c-ee80-4f78-9c40-d60b9a515487" containerID="a7e001a58e0c30838a7e5829a0b6beff57ec3cedebcf19953fc6b49946dc6493" exitCode=1
Apr 16 17:27:27.779339 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.779296    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh" event={"ID":"432c565c-ee80-4f78-9c40-d60b9a515487","Type":"ContainerDied","Data":"a7e001a58e0c30838a7e5829a0b6beff57ec3cedebcf19953fc6b49946dc6493"}
Apr 16 17:27:27.779339 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.779319    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh" event={"ID":"432c565c-ee80-4f78-9c40-d60b9a515487","Type":"ContainerDied","Data":"a865def94b7c90b31a391bb0e002bffdc08d86bcac73f7e21590dd2a4f500000"}
Apr 16 17:27:27.779339 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.779333    2578 scope.go:117] "RemoveContainer" containerID="a7e001a58e0c30838a7e5829a0b6beff57ec3cedebcf19953fc6b49946dc6493"
Apr 16 17:27:27.779339 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.779336    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh"
Apr 16 17:27:27.792131 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.792109    2578 scope.go:117] "RemoveContainer" containerID="6ff215ba394ceb90b3868093d1ee00a4ed5d53cee4f3ac5330cba5c021460483"
Apr 16 17:27:27.798945 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.798930    2578 scope.go:117] "RemoveContainer" containerID="a7e001a58e0c30838a7e5829a0b6beff57ec3cedebcf19953fc6b49946dc6493"
Apr 16 17:27:27.799196 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:27:27.799178    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e001a58e0c30838a7e5829a0b6beff57ec3cedebcf19953fc6b49946dc6493\": container with ID starting with a7e001a58e0c30838a7e5829a0b6beff57ec3cedebcf19953fc6b49946dc6493 not found: ID does not exist" containerID="a7e001a58e0c30838a7e5829a0b6beff57ec3cedebcf19953fc6b49946dc6493"
Apr 16 17:27:27.799244 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.799209    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e001a58e0c30838a7e5829a0b6beff57ec3cedebcf19953fc6b49946dc6493"} err="failed to get container status \"a7e001a58e0c30838a7e5829a0b6beff57ec3cedebcf19953fc6b49946dc6493\": rpc error: code = NotFound desc = could not find container \"a7e001a58e0c30838a7e5829a0b6beff57ec3cedebcf19953fc6b49946dc6493\": container with ID starting with a7e001a58e0c30838a7e5829a0b6beff57ec3cedebcf19953fc6b49946dc6493 not found: ID does not exist"
Apr 16 17:27:27.799244 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.799227    2578 scope.go:117] "RemoveContainer" containerID="6ff215ba394ceb90b3868093d1ee00a4ed5d53cee4f3ac5330cba5c021460483"
Apr 16 17:27:27.799423 ip-10-0-140-164 kubenswrapper[2578]: E0416 17:27:27.799408    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff215ba394ceb90b3868093d1ee00a4ed5d53cee4f3ac5330cba5c021460483\": container with ID starting with 6ff215ba394ceb90b3868093d1ee00a4ed5d53cee4f3ac5330cba5c021460483 not found: ID does not exist" containerID="6ff215ba394ceb90b3868093d1ee00a4ed5d53cee4f3ac5330cba5c021460483"
Apr 16 17:27:27.799471 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.799425    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff215ba394ceb90b3868093d1ee00a4ed5d53cee4f3ac5330cba5c021460483"} err="failed to get container status \"6ff215ba394ceb90b3868093d1ee00a4ed5d53cee4f3ac5330cba5c021460483\": rpc error: code = NotFound desc = could not find container \"6ff215ba394ceb90b3868093d1ee00a4ed5d53cee4f3ac5330cba5c021460483\": container with ID starting with 6ff215ba394ceb90b3868093d1ee00a4ed5d53cee4f3ac5330cba5c021460483 not found: ID does not exist"
Apr 16 17:27:27.808728 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.808701    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh"]
Apr 16 17:27:27.811938 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.811919    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-57b648fdbb-h28vh"]
Apr 16 17:27:27.825807 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.825787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7292b411-b4b5-4db8-bc2e-7fec6a7160de-must-gather-output\") pod \"must-gather-p265v\" (UID: \"7292b411-b4b5-4db8-bc2e-7fec6a7160de\") " pod="openshift-must-gather-z2hj7/must-gather-p265v"
Apr 16 17:27:27.825891 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.825849    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpnfh\" (UniqueName: \"kubernetes.io/projected/7292b411-b4b5-4db8-bc2e-7fec6a7160de-kube-api-access-hpnfh\") pod \"must-gather-p265v\" (UID: \"7292b411-b4b5-4db8-bc2e-7fec6a7160de\") " pod="openshift-must-gather-z2hj7/must-gather-p265v"
Apr 16 17:27:27.926602 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.926581    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hpnfh\" (UniqueName: \"kubernetes.io/projected/7292b411-b4b5-4db8-bc2e-7fec6a7160de-kube-api-access-hpnfh\") pod \"must-gather-p265v\" (UID: \"7292b411-b4b5-4db8-bc2e-7fec6a7160de\") " pod="openshift-must-gather-z2hj7/must-gather-p265v"
Apr 16 17:27:27.926680 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.926622    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7292b411-b4b5-4db8-bc2e-7fec6a7160de-must-gather-output\") pod \"must-gather-p265v\" (UID: \"7292b411-b4b5-4db8-bc2e-7fec6a7160de\") " pod="openshift-must-gather-z2hj7/must-gather-p265v"
Apr 16 17:27:27.926892 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.926878    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7292b411-b4b5-4db8-bc2e-7fec6a7160de-must-gather-output\") pod \"must-gather-p265v\" (UID: \"7292b411-b4b5-4db8-bc2e-7fec6a7160de\") " pod="openshift-must-gather-z2hj7/must-gather-p265v"
Apr 16 17:27:27.934062 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:27.934036    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpnfh\" (UniqueName: \"kubernetes.io/projected/7292b411-b4b5-4db8-bc2e-7fec6a7160de-kube-api-access-hpnfh\") pod \"must-gather-p265v\" (UID: \"7292b411-b4b5-4db8-bc2e-7fec6a7160de\") " pod="openshift-must-gather-z2hj7/must-gather-p265v"
Apr 16 17:27:28.022239 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:28.022191    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z2hj7/must-gather-p265v"
Apr 16 17:27:28.131444 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:28.131128    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z2hj7/must-gather-p265v"]
Apr 16 17:27:28.133777 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:27:28.133751    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7292b411_b4b5_4db8_bc2e_7fec6a7160de.slice/crio-598b13d7fab1fded5689c52f3becbfca25d173b063e961f7accb3440cae82a4c WatchSource:0}: Error finding container 598b13d7fab1fded5689c52f3becbfca25d173b063e961f7accb3440cae82a4c: Status 404 returned error can't find the container with id 598b13d7fab1fded5689c52f3becbfca25d173b063e961f7accb3440cae82a4c
Apr 16 17:27:28.783646 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:28.783610    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z2hj7/must-gather-p265v" event={"ID":"7292b411-b4b5-4db8-bc2e-7fec6a7160de","Type":"ContainerStarted","Data":"598b13d7fab1fded5689c52f3becbfca25d173b063e961f7accb3440cae82a4c"}
Apr 16 17:27:29.235988 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:29.235953    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="432c565c-ee80-4f78-9c40-d60b9a515487" path="/var/lib/kubelet/pods/432c565c-ee80-4f78-9c40-d60b9a515487/volumes"
Apr 16 17:27:33.803079 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:33.803039    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z2hj7/must-gather-p265v" event={"ID":"7292b411-b4b5-4db8-bc2e-7fec6a7160de","Type":"ContainerStarted","Data":"c31eff94382a26b40a128d5f6d46f0c094aba9ca966833c880a053e0ba3bfb96"}
Apr 16 17:27:33.803079 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:33.803080    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z2hj7/must-gather-p265v" event={"ID":"7292b411-b4b5-4db8-bc2e-7fec6a7160de","Type":"ContainerStarted","Data":"539031d24f8b71f52ff0b9f0036175f11ada09be252b6c17a2444b463b792d34"}
Apr 16 17:27:33.818125 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:33.818079    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-z2hj7/must-gather-p265v" podStartSLOduration=2.0945419 podStartE2EDuration="6.818064489s" podCreationTimestamp="2026-04-16 17:27:27 +0000 UTC" firstStartedPulling="2026-04-16 17:27:28.135389301 +0000 UTC m=+3457.532605668" lastFinishedPulling="2026-04-16 17:27:32.858911888 +0000 UTC m=+3462.256128257" observedRunningTime="2026-04-16 17:27:33.816511621 +0000 UTC m=+3463.213728010" watchObservedRunningTime="2026-04-16 17:27:33.818064489 +0000 UTC m=+3463.215280877"
Apr 16 17:27:53.864598 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:53.864549    2578 generic.go:358] "Generic (PLEG): container finished" podID="7292b411-b4b5-4db8-bc2e-7fec6a7160de" containerID="539031d24f8b71f52ff0b9f0036175f11ada09be252b6c17a2444b463b792d34" exitCode=0
Apr 16 17:27:53.864996 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:53.864623    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z2hj7/must-gather-p265v" event={"ID":"7292b411-b4b5-4db8-bc2e-7fec6a7160de","Type":"ContainerDied","Data":"539031d24f8b71f52ff0b9f0036175f11ada09be252b6c17a2444b463b792d34"}
Apr 16 17:27:53.864996 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:53.864904    2578 scope.go:117] "RemoveContainer" containerID="539031d24f8b71f52ff0b9f0036175f11ada09be252b6c17a2444b463b792d34"
Apr 16 17:27:54.224475 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:54.224444    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-z2hj7_must-gather-p265v_7292b411-b4b5-4db8-bc2e-7fec6a7160de/gather/0.log"
Apr 16 17:27:54.855298 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:54.855262    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qzsql/must-gather-54ccw"]
Apr 16 17:27:54.862032 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:54.859228    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qzsql/must-gather-54ccw"
Apr 16 17:27:54.862432 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:54.862385    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qzsql\"/\"kube-root-ca.crt\""
Apr 16 17:27:54.863197 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:54.863173    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qzsql\"/\"openshift-service-ca.crt\""
Apr 16 17:27:54.863335 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:54.863209    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qzsql\"/\"default-dockercfg-vd8tg\""
Apr 16 17:27:54.865666 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:54.865647    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qzsql/must-gather-54ccw"]
Apr 16 17:27:54.952405 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:54.952366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/08e67373-b261-44ec-b465-bd6550a23079-must-gather-output\") pod \"must-gather-54ccw\" (UID: \"08e67373-b261-44ec-b465-bd6550a23079\") " pod="openshift-must-gather-qzsql/must-gather-54ccw"
Apr 16 17:27:54.952568 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:54.952417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7t7l\" (UniqueName: \"kubernetes.io/projected/08e67373-b261-44ec-b465-bd6550a23079-kube-api-access-c7t7l\") pod \"must-gather-54ccw\" (UID: \"08e67373-b261-44ec-b465-bd6550a23079\") " pod="openshift-must-gather-qzsql/must-gather-54ccw"
Apr 16 17:27:55.053815 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:55.053784    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/08e67373-b261-44ec-b465-bd6550a23079-must-gather-output\") pod \"must-gather-54ccw\" (UID: \"08e67373-b261-44ec-b465-bd6550a23079\") " pod="openshift-must-gather-qzsql/must-gather-54ccw"
Apr 16 17:27:55.053970 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:55.053834    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c7t7l\" (UniqueName: \"kubernetes.io/projected/08e67373-b261-44ec-b465-bd6550a23079-kube-api-access-c7t7l\") pod \"must-gather-54ccw\" (UID: \"08e67373-b261-44ec-b465-bd6550a23079\") " pod="openshift-must-gather-qzsql/must-gather-54ccw"
Apr 16 17:27:55.054202 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:55.054179    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/08e67373-b261-44ec-b465-bd6550a23079-must-gather-output\") pod \"must-gather-54ccw\" (UID: \"08e67373-b261-44ec-b465-bd6550a23079\") " pod="openshift-must-gather-qzsql/must-gather-54ccw"
Apr 16 17:27:55.061550 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:55.061524    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7t7l\" (UniqueName: \"kubernetes.io/projected/08e67373-b261-44ec-b465-bd6550a23079-kube-api-access-c7t7l\") pod \"must-gather-54ccw\" (UID: \"08e67373-b261-44ec-b465-bd6550a23079\") " pod="openshift-must-gather-qzsql/must-gather-54ccw"
Apr 16 17:27:55.173242 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:55.173173    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qzsql/must-gather-54ccw"
Apr 16 17:27:55.288855 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:55.288823    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qzsql/must-gather-54ccw"]
Apr 16 17:27:55.291822 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:27:55.291796    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08e67373_b261_44ec_b465_bd6550a23079.slice/crio-56e2704a560679b3e298eb46e567b51481ef432a152238c848b6ef787367d1d1 WatchSource:0}: Error finding container 56e2704a560679b3e298eb46e567b51481ef432a152238c848b6ef787367d1d1: Status 404 returned error can't find the container with id 56e2704a560679b3e298eb46e567b51481ef432a152238c848b6ef787367d1d1
Apr 16 17:27:55.872286 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:55.872244    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qzsql/must-gather-54ccw" event={"ID":"08e67373-b261-44ec-b465-bd6550a23079","Type":"ContainerStarted","Data":"56e2704a560679b3e298eb46e567b51481ef432a152238c848b6ef787367d1d1"}
Apr 16 17:27:56.877610 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:56.877577    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qzsql/must-gather-54ccw" event={"ID":"08e67373-b261-44ec-b465-bd6550a23079","Type":"ContainerStarted","Data":"dbd31a28b9644e11f306b0de861af6280996ec56966b2511bd6df39697fd88a3"}
Apr 16 17:27:56.877929 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:56.877618    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qzsql/must-gather-54ccw" event={"ID":"08e67373-b261-44ec-b465-bd6550a23079","Type":"ContainerStarted","Data":"fdfc88ba9706790f5d8ebc56ee4f397b106909090cfa59fb987a9ee7ee7ef07f"}
Apr 16 17:27:56.893674 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:56.893627    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-qzsql/must-gather-54ccw" podStartSLOduration=2.037395274 podStartE2EDuration="2.89361517s" podCreationTimestamp="2026-04-16 17:27:54 +0000 UTC" firstStartedPulling="2026-04-16 17:27:55.293549404 +0000 UTC m=+3484.690765770" lastFinishedPulling="2026-04-16 17:27:56.149769299 +0000 UTC m=+3485.546985666" observedRunningTime="2026-04-16 17:27:56.891432027 +0000 UTC m=+3486.288648415" watchObservedRunningTime="2026-04-16 17:27:56.89361517 +0000 UTC m=+3486.290831559"
Apr 16 17:27:59.680321 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:59.680233    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bk4tn_1bb21574-8866-4f9f-a9e3-9fb3e776094e/global-pull-secret-syncer/0.log"
Apr 16 17:27:59.693984 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:59.693954    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-z2hj7/must-gather-p265v"]
Apr 16 17:27:59.694167 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:59.694148    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-z2hj7/must-gather-p265v" podUID="7292b411-b4b5-4db8-bc2e-7fec6a7160de" containerName="copy" containerID="cri-o://c31eff94382a26b40a128d5f6d46f0c094aba9ca966833c880a053e0ba3bfb96" gracePeriod=2
Apr 16 17:27:59.696748 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:59.696709    2578 status_manager.go:895] "Failed to get status for pod" podUID="7292b411-b4b5-4db8-bc2e-7fec6a7160de" pod="openshift-must-gather-z2hj7/must-gather-p265v" err="pods \"must-gather-p265v\" is forbidden: User \"system:node:ip-10-0-140-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-z2hj7\": no relationship found between node 'ip-10-0-140-164.ec2.internal' and this object"
Apr 16 17:27:59.697358 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:59.697335    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-z2hj7/must-gather-p265v"]
Apr 16 17:27:59.868878 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:59.868826    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rpnzb_350fba86-71d9-4d96-b9ed-de22ab9333c9/konnectivity-agent/0.log"
Apr 16 17:27:59.899350 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:59.899320    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-z2hj7_must-gather-p265v_7292b411-b4b5-4db8-bc2e-7fec6a7160de/copy/0.log"
Apr 16 17:27:59.899729 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:59.899704    2578 generic.go:358] "Generic (PLEG): container finished" podID="7292b411-b4b5-4db8-bc2e-7fec6a7160de" containerID="c31eff94382a26b40a128d5f6d46f0c094aba9ca966833c880a053e0ba3bfb96" exitCode=143
Apr 16 17:27:59.941946 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:59.941890    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-z2hj7_must-gather-p265v_7292b411-b4b5-4db8-bc2e-7fec6a7160de/copy/0.log"
Apr 16 17:27:59.942281 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:59.942264    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z2hj7/must-gather-p265v"
Apr 16 17:27:59.944215 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:27:59.944187    2578 status_manager.go:895] "Failed to get status for pod" podUID="7292b411-b4b5-4db8-bc2e-7fec6a7160de" pod="openshift-must-gather-z2hj7/must-gather-p265v" err="pods \"must-gather-p265v\" is forbidden: User \"system:node:ip-10-0-140-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-z2hj7\": no relationship found between node 'ip-10-0-140-164.ec2.internal' and this object"
Apr 16 17:28:00.005803 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:00.005766    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-164.ec2.internal_0a4ffc4283ad098ce5617bb701ba0817/haproxy/0.log"
Apr 16 17:28:00.097989 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:00.097946    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpnfh\" (UniqueName: \"kubernetes.io/projected/7292b411-b4b5-4db8-bc2e-7fec6a7160de-kube-api-access-hpnfh\") pod \"7292b411-b4b5-4db8-bc2e-7fec6a7160de\" (UID: \"7292b411-b4b5-4db8-bc2e-7fec6a7160de\") "
Apr 16 17:28:00.098180 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:00.098034    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7292b411-b4b5-4db8-bc2e-7fec6a7160de-must-gather-output\") pod \"7292b411-b4b5-4db8-bc2e-7fec6a7160de\" (UID: \"7292b411-b4b5-4db8-bc2e-7fec6a7160de\") "
Apr 16 17:28:00.099342 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:00.099311    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7292b411-b4b5-4db8-bc2e-7fec6a7160de-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "7292b411-b4b5-4db8-bc2e-7fec6a7160de" (UID: "7292b411-b4b5-4db8-bc2e-7fec6a7160de"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 17:28:00.100698 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:00.100666    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7292b411-b4b5-4db8-bc2e-7fec6a7160de-kube-api-access-hpnfh" (OuterVolumeSpecName: "kube-api-access-hpnfh") pod "7292b411-b4b5-4db8-bc2e-7fec6a7160de" (UID: "7292b411-b4b5-4db8-bc2e-7fec6a7160de"). InnerVolumeSpecName "kube-api-access-hpnfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 17:28:00.199502 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:00.199472    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpnfh\" (UniqueName: \"kubernetes.io/projected/7292b411-b4b5-4db8-bc2e-7fec6a7160de-kube-api-access-hpnfh\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:28:00.199502 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:00.199501    2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7292b411-b4b5-4db8-bc2e-7fec6a7160de-must-gather-output\") on node \"ip-10-0-140-164.ec2.internal\" DevicePath \"\""
Apr 16 17:28:00.906076 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:00.906035    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-z2hj7_must-gather-p265v_7292b411-b4b5-4db8-bc2e-7fec6a7160de/copy/0.log"
Apr 16 17:28:00.906662 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:00.906636    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z2hj7/must-gather-p265v"
Apr 16 17:28:00.910601 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:00.906614    2578 scope.go:117] "RemoveContainer" containerID="c31eff94382a26b40a128d5f6d46f0c094aba9ca966833c880a053e0ba3bfb96"
Apr 16 17:28:00.913423 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:00.913393    2578 status_manager.go:895] "Failed to get status for pod" podUID="7292b411-b4b5-4db8-bc2e-7fec6a7160de" pod="openshift-must-gather-z2hj7/must-gather-p265v" err="pods \"must-gather-p265v\" is forbidden: User \"system:node:ip-10-0-140-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-z2hj7\": no relationship found between node 'ip-10-0-140-164.ec2.internal' and this object"
Apr 16 17:28:00.926011 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:00.925992    2578 scope.go:117] "RemoveContainer" containerID="539031d24f8b71f52ff0b9f0036175f11ada09be252b6c17a2444b463b792d34"
Apr 16 17:28:00.929895 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:00.929866    2578 status_manager.go:895] "Failed to get status for pod" podUID="7292b411-b4b5-4db8-bc2e-7fec6a7160de" pod="openshift-must-gather-z2hj7/must-gather-p265v" err="pods \"must-gather-p265v\" is forbidden: User \"system:node:ip-10-0-140-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-z2hj7\": no relationship found between node 'ip-10-0-140-164.ec2.internal' and this object"
Apr 16 17:28:01.239592 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:01.239536    2578 status_manager.go:895] "Failed to get status for pod" podUID="7292b411-b4b5-4db8-bc2e-7fec6a7160de" pod="openshift-must-gather-z2hj7/must-gather-p265v" err="pods \"must-gather-p265v\" is forbidden: User \"system:node:ip-10-0-140-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-z2hj7\": no relationship found between node 'ip-10-0-140-164.ec2.internal' and this object"
Apr 16 17:28:01.240482 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:01.240068    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7292b411-b4b5-4db8-bc2e-7fec6a7160de" path="/var/lib/kubelet/pods/7292b411-b4b5-4db8-bc2e-7fec6a7160de/volumes"
Apr 16 17:28:03.346022 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.345971    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6mx4_f076b615-76d7-4867-ae84-0374653e85eb/node-exporter/0.log"
Apr 16 17:28:03.365598 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.365466    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6mx4_f076b615-76d7-4867-ae84-0374653e85eb/kube-rbac-proxy/0.log"
Apr 16 17:28:03.389782 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.389730    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6mx4_f076b615-76d7-4867-ae84-0374653e85eb/init-textfile/0.log"
Apr 16 17:28:03.417535 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.417505    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-5669946b84-cjq55_ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a/kube-rbac-proxy-main/0.log"
Apr 16 17:28:03.438122 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.438081    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-5669946b84-cjq55_ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a/kube-rbac-proxy-self/0.log"
Apr 16 17:28:03.463787 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.463749    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-5669946b84-cjq55_ebca9f02-a170-4c8c-a673-c8bd7d3c7b5a/openshift-state-metrics/0.log"
Apr 16 17:28:03.524739 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.524678    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed3d8c6d-3bd0-4cc6-beb9-c5e737944966/prometheus/0.log"
Apr 16 17:28:03.544374 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.544338    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed3d8c6d-3bd0-4cc6-beb9-c5e737944966/config-reloader/0.log"
Apr 16 17:28:03.565212 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.565128    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed3d8c6d-3bd0-4cc6-beb9-c5e737944966/thanos-sidecar/0.log"
Apr 16 17:28:03.586084 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.586055    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed3d8c6d-3bd0-4cc6-beb9-c5e737944966/kube-rbac-proxy-web/0.log"
Apr 16 17:28:03.608067 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.608038    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed3d8c6d-3bd0-4cc6-beb9-c5e737944966/kube-rbac-proxy/0.log"
Apr 16 17:28:03.627089 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.627064    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed3d8c6d-3bd0-4cc6-beb9-c5e737944966/kube-rbac-proxy-thanos/0.log"
Apr 16 17:28:03.647660 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.647615    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed3d8c6d-3bd0-4cc6-beb9-c5e737944966/init-config-reloader/0.log"
Apr 16 17:28:03.677330 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.677293    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-78f957474d-lbm2d_c617adf1-5e8f-4c9c-8755-540fc1fcb407/prometheus-operator/0.log"
Apr 16 17:28:03.694685 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.694654    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-78f957474d-lbm2d_c617adf1-5e8f-4c9c-8755-540fc1fcb407/kube-rbac-proxy/0.log"
Apr 16 17:28:03.716397 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.716371    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-9cb97cd87-gh2cl_4388c084-9228-49d3-ac5a-e5e0951c6e83/prometheus-operator-admission-webhook/0.log"
Apr 16 17:28:03.818887 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.818794    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6947b544f5-f4vb2_351052b4-54f5-459b-8b1e-4542e7b6d41d/thanos-query/0.log"
Apr 16 17:28:03.837650 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.837614    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6947b544f5-f4vb2_351052b4-54f5-459b-8b1e-4542e7b6d41d/kube-rbac-proxy-web/0.log"
Apr 16 17:28:03.859969 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.859943    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6947b544f5-f4vb2_351052b4-54f5-459b-8b1e-4542e7b6d41d/kube-rbac-proxy/0.log"
Apr 16 17:28:03.878799 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.878764    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6947b544f5-f4vb2_351052b4-54f5-459b-8b1e-4542e7b6d41d/prom-label-proxy/0.log"
Apr 16 17:28:03.899109 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.899085    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6947b544f5-f4vb2_351052b4-54f5-459b-8b1e-4542e7b6d41d/kube-rbac-proxy-rules/0.log"
Apr 16 17:28:03.918276 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:03.918251    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6947b544f5-f4vb2_351052b4-54f5-459b-8b1e-4542e7b6d41d/kube-rbac-proxy-metrics/0.log"
Apr 16 17:28:05.965957 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:05.965918    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d44c76d5d-74zpg_89e9c8a0-9fc5-4d46-aaa4-bcdbe6257208/console/0.log"
Apr 16 17:28:06.008023 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:06.007993    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-586b57c7b4-fxvzp_b99ebca1-5a6b-44b9-8db2-e29fad719f8b/download-server/0.log"
Apr 16 17:28:07.067886 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.067855    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jrfnm_f43bc80a-5182-4784-9565-41fe3bf16f60/dns/0.log"
Apr 16 17:28:07.086188 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.086155    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jrfnm_f43bc80a-5182-4784-9565-41fe3bf16f60/kube-rbac-proxy/0.log"
Apr 16 17:28:07.170528 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.170496    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fbgtp_12c1b394-8723-4774-aecd-3c20c35fc722/dns-node-resolver/0.log"
Apr 16 17:28:07.209676 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.209650    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"]
Apr 16 17:28:07.209976 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.209965    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7292b411-b4b5-4db8-bc2e-7fec6a7160de" containerName="gather"
Apr 16 17:28:07.210024 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.209978    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7292b411-b4b5-4db8-bc2e-7fec6a7160de" containerName="gather"
Apr 16 17:28:07.210024 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.209988    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7292b411-b4b5-4db8-bc2e-7fec6a7160de" containerName="copy"
Apr 16 17:28:07.210024 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.209993    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7292b411-b4b5-4db8-bc2e-7fec6a7160de" containerName="copy"
Apr 16 17:28:07.210118 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.210056    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="7292b411-b4b5-4db8-bc2e-7fec6a7160de" containerName="copy"
Apr 16 17:28:07.210118 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.210069    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="7292b411-b4b5-4db8-bc2e-7fec6a7160de" containerName="gather"
Apr 16 17:28:07.214390 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.214367    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.219152 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.219129    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"]
Apr 16 17:28:07.263475 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.263443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b777f908-786f-48c4-8719-ce38296acfac-sys\") pod \"perf-node-gather-daemonset-6c4kc\" (UID: \"b777f908-786f-48c4-8719-ce38296acfac\") " pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.263680 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.263482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b777f908-786f-48c4-8719-ce38296acfac-proc\") pod \"perf-node-gather-daemonset-6c4kc\" (UID: \"b777f908-786f-48c4-8719-ce38296acfac\") " pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.263680 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.263531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b777f908-786f-48c4-8719-ce38296acfac-podres\") pod \"perf-node-gather-daemonset-6c4kc\" (UID: \"b777f908-786f-48c4-8719-ce38296acfac\") " pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.263680 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.263580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b777f908-786f-48c4-8719-ce38296acfac-lib-modules\") pod \"perf-node-gather-daemonset-6c4kc\" (UID: \"b777f908-786f-48c4-8719-ce38296acfac\") " pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.263680 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.263630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f79b\" (UniqueName: \"kubernetes.io/projected/b777f908-786f-48c4-8719-ce38296acfac-kube-api-access-9f79b\") pod \"perf-node-gather-daemonset-6c4kc\" (UID: \"b777f908-786f-48c4-8719-ce38296acfac\") " pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.364942 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.364857    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9f79b\" (UniqueName: \"kubernetes.io/projected/b777f908-786f-48c4-8719-ce38296acfac-kube-api-access-9f79b\") pod \"perf-node-gather-daemonset-6c4kc\" (UID: \"b777f908-786f-48c4-8719-ce38296acfac\") " pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.365103 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.364947    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b777f908-786f-48c4-8719-ce38296acfac-sys\") pod \"perf-node-gather-daemonset-6c4kc\" (UID: \"b777f908-786f-48c4-8719-ce38296acfac\") " pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.365103 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.364982    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b777f908-786f-48c4-8719-ce38296acfac-proc\") pod \"perf-node-gather-daemonset-6c4kc\" (UID: \"b777f908-786f-48c4-8719-ce38296acfac\") " pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.365103 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.365013    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b777f908-786f-48c4-8719-ce38296acfac-podres\") pod \"perf-node-gather-daemonset-6c4kc\" (UID: \"b777f908-786f-48c4-8719-ce38296acfac\") " pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.365103 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.365043    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b777f908-786f-48c4-8719-ce38296acfac-lib-modules\") pod \"perf-node-gather-daemonset-6c4kc\" (UID: \"b777f908-786f-48c4-8719-ce38296acfac\") " pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.365103 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.365068    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b777f908-786f-48c4-8719-ce38296acfac-sys\") pod \"perf-node-gather-daemonset-6c4kc\" (UID: \"b777f908-786f-48c4-8719-ce38296acfac\") " pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.365103 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.365081    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b777f908-786f-48c4-8719-ce38296acfac-proc\") pod \"perf-node-gather-daemonset-6c4kc\" (UID: \"b777f908-786f-48c4-8719-ce38296acfac\") " pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.365357 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.365168    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b777f908-786f-48c4-8719-ce38296acfac-podres\") pod \"perf-node-gather-daemonset-6c4kc\" (UID: \"b777f908-786f-48c4-8719-ce38296acfac\") " pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.365357 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.365185    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b777f908-786f-48c4-8719-ce38296acfac-lib-modules\") pod \"perf-node-gather-daemonset-6c4kc\" (UID: \"b777f908-786f-48c4-8719-ce38296acfac\") " pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.373250 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.373221    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f79b\" (UniqueName: \"kubernetes.io/projected/b777f908-786f-48c4-8719-ce38296acfac-kube-api-access-9f79b\") pod \"perf-node-gather-daemonset-6c4kc\" (UID: \"b777f908-786f-48c4-8719-ce38296acfac\") " pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.526413 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.526381    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.656254 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.656223    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"]
Apr 16 17:28:07.658848 ip-10-0-140-164 kubenswrapper[2578]: W0416 17:28:07.658818    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb777f908_786f_48c4_8719_ce38296acfac.slice/crio-2626ad28e282dd4023ac9bc895c61e6a51b59ee9e119d003bf48e792c4825004 WatchSource:0}: Error finding container 2626ad28e282dd4023ac9bc895c61e6a51b59ee9e119d003bf48e792c4825004: Status 404 returned error can't find the container with id 2626ad28e282dd4023ac9bc895c61e6a51b59ee9e119d003bf48e792c4825004
Apr 16 17:28:07.666437 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.666406    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7qft9_d36c965f-65b5-4340-8151-e2ff609581b6/node-ca/0.log"
Apr 16 17:28:07.936426 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.936349    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc" event={"ID":"b777f908-786f-48c4-8719-ce38296acfac","Type":"ContainerStarted","Data":"db4a4d20faa2164832def944fa1687ce6301871aee0ebb1ce539a9e46e4cffac"}
Apr 16 17:28:07.936426 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.936386    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc" event={"ID":"b777f908-786f-48c4-8719-ce38296acfac","Type":"ContainerStarted","Data":"2626ad28e282dd4023ac9bc895c61e6a51b59ee9e119d003bf48e792c4825004"}
Apr 16 17:28:07.936685 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.936469    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:07.950518 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:07.950465    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc" podStartSLOduration=0.950448973 podStartE2EDuration="950.448973ms" podCreationTimestamp="2026-04-16 17:28:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:28:07.949684473 +0000 UTC m=+3497.346900875" watchObservedRunningTime="2026-04-16 17:28:07.950448973 +0000 UTC m=+3497.347665363"
Apr 16 17:28:08.743858 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:08.743823    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mvjtr_b569a454-1fb0-4aae-ace9-f8ea8f122839/serve-healthcheck-canary/0.log"
Apr 16 17:28:09.057685 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:09.057604    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j72gz_9d15723d-383a-4598-907c-bb0f4224279d/kube-rbac-proxy/0.log"
Apr 16 17:28:09.074985 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:09.074959    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j72gz_9d15723d-383a-4598-907c-bb0f4224279d/exporter/0.log"
Apr 16 17:28:09.094827 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:09.094798    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j72gz_9d15723d-383a-4598-907c-bb0f4224279d/extractor/0.log"
Apr 16 17:28:11.359670 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:11.359644    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-l86r8_097dd376-9dd2-4b76-8a0e-16f5ec05e180/server/0.log"
Apr 16 17:28:11.751075 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:11.751038    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-wxvfp_b8d03003-d679-4684-b987-75a4ec1a678f/s3-init/0.log"
Apr 16 17:28:11.771598 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:11.771570    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-bb56r_4db2d7a6-8e21-4163-bae8-85e0aad131a5/s3-tls-init-custom/0.log"
Apr 16 17:28:11.819481 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:11.819453    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-4pgdf_9926b7e5-4866-41e3-b468-3459dd74ea36/s3-tls-init-serving/0.log"
Apr 16 17:28:11.866817 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:11.866784    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-5c88b85bb7-4dfxv_627c8c08-25bd-4e9f-9b1b-1baf6c101d30/seaweedfs-tls-custom/0.log"
Apr 16 17:28:13.952103 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:13.952071    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qzsql/perf-node-gather-daemonset-6c4kc"
Apr 16 17:28:16.824567 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:16.824520    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7cqz2_c5d3493d-e664-4bec-a956-08fc7c9bbf03/kube-multus/0.log"
Apr 16 17:28:17.184918 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:17.184843    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnptn_57674e7d-ee63-4a14-85d1-8c35f765106d/kube-multus-additional-cni-plugins/0.log"
Apr 16 17:28:17.207272 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:17.207243    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnptn_57674e7d-ee63-4a14-85d1-8c35f765106d/egress-router-binary-copy/0.log"
Apr 16 17:28:17.227893 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:17.227863    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnptn_57674e7d-ee63-4a14-85d1-8c35f765106d/cni-plugins/0.log"
Apr 16 17:28:17.247335 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:17.247311    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnptn_57674e7d-ee63-4a14-85d1-8c35f765106d/bond-cni-plugin/0.log"
Apr 16 17:28:17.268707 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:17.268680    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnptn_57674e7d-ee63-4a14-85d1-8c35f765106d/routeoverride-cni/0.log"
Apr 16 17:28:17.288592 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:17.288543    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnptn_57674e7d-ee63-4a14-85d1-8c35f765106d/whereabouts-cni-bincopy/0.log"
Apr 16 17:28:17.354439 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:17.354410    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnptn_57674e7d-ee63-4a14-85d1-8c35f765106d/whereabouts-cni/0.log"
Apr 16 17:28:17.488953 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:17.488927    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wq65n_21a9695d-812c-4ad6-9b18-ade579b343e0/network-metrics-daemon/0.log"
Apr 16 17:28:17.506423 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:17.506383    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wq65n_21a9695d-812c-4ad6-9b18-ade579b343e0/kube-rbac-proxy/0.log"
Apr 16 17:28:18.797083 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:18.797052    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-controller/0.log"
Apr 16 17:28:18.811455 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:18.811416    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/0.log"
Apr 16 17:28:18.847107 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:18.847086    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovn-acl-logging/1.log"
Apr 16 17:28:18.862624 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:18.862605    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/kube-rbac-proxy-node/0.log"
Apr 16 17:28:18.880948 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:18.880929    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/kube-rbac-proxy-ovn-metrics/0.log"
Apr 16 17:28:18.900787 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:18.900758    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/northd/0.log"
Apr 16 17:28:18.918616 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:18.918601    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/nbdb/0.log"
Apr 16 17:28:18.944677 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:18.944657    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/sbdb/0.log"
Apr 16 17:28:19.151278 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:19.151204    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wd8ld_78b0d7e4-bab7-4f8d-ac1e-f45bede95414/ovnkube-controller/0.log"
Apr 16 17:28:20.075060 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:20.075029    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-w2mqw_29bd11bf-6683-4b59-9b71-e5f556a0ad04/network-check-target-container/0.log"
Apr 16 17:28:20.931826 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:20.931797    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-xdwz4_d64db671-44fb-4c47-bda6-b8c720247f41/iptables-alerter/0.log"
Apr 16 17:28:21.593516 ip-10-0-140-164 kubenswrapper[2578]: I0416 17:28:21.593476    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-pdmjc_f98643b4-a68c-4a82-ac48-50ebf25135ba/tuned/0.log"